[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:19:40 dist-test-slave-6ntk krb5kdc[12351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:19:40 dist-test-slave-6ntk krb5kdc[12351](info): set up 2 sockets
Aug 08 08:19:40 dist-test-slave-6ntk krb5kdc[12351](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 08:19:41.209224 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:19:41 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:19:41.358740 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33861
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:41.491612 12370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:41.491977 12370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:41.492069 12370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:41.496174 12370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:19:41.496284 12370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:41.496325 12370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:41.496361 12370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:19:41.496398 12370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:19:41.502054 12370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:41.503800 12370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:41.505192 12370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:41.520815 12377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:41.521036 12370 server_base.cc:1034] running on GCE node
W20260808 08:19:41.523010 12380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:41.523074 12378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:41.524211 12370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:41.526288 12370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:41.527555 12370 hybrid_clock.cc:648] HybridClock initialized: now 1786177181527513 us; error 79 us; skew 500 ppm
Aug 08 08:19:41 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177181, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:41.530969 12370 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:19:41.532380 12370 webserver.cc:460] Webserver started at http://127.12.13.62:34563/ using document root <none> and password file <none>
I20260808 08:19:41.533340 12370 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:41.533458 12370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:41.533929 12370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:41.536811 12370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "8b47afd0c64c4d399470a7fff7740b3c"
format_stamp: "Formatted at 2026-08-08 08:19:41 on dist-test-slave-6ntk"
server_key: "fbad6b15542867524f668beefbeb4c63"
server_key_iv: "a25f18f11b487e37911bb2a50c01ef94"
server_key_version: "encryptionkey@0"
I20260808 08:19:41.537544 12370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "8b47afd0c64c4d399470a7fff7740b3c"
format_stamp: "Formatted at 2026-08-08 08:19:41 on dist-test-slave-6ntk"
server_key: "fbad6b15542867524f668beefbeb4c63"
server_key_iv: "a25f18f11b487e37911bb2a50c01ef94"
server_key_version: "encryptionkey@0"
I20260808 08:19:41.542951 12370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 08:19:41.547282 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:41.549084 12370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:19:41.549333 12370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "8b47afd0c64c4d399470a7fff7740b3c"
format_stamp: "Formatted at 2026-08-08 08:19:41 on dist-test-slave-6ntk"
server_key: "fbad6b15542867524f668beefbeb4c63"
server_key_iv: "a25f18f11b487e37911bb2a50c01ef94"
server_key_version: "encryptionkey@0"
I20260808 08:19:41.549485 12370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:41.575827 12370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:41.580035 12370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:41.580361 12370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:41.597005 12370 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:33861
I20260808 08:19:41.597215 12465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:33861 every 8 connection(s)
I20260808 08:19:41.598495 12370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:19:41.603839 12466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:41.607786 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12370
I20260808 08:19:41.607986 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:19:41.609472 12340 external_mini_cluster.cc:1428] Setting key d187413f7e024d78654ca1c4d1c16649
I20260808 08:19:41.612659 12466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c: Bootstrap starting.
I20260808 08:19:41.617389 12466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:41.619056 12466 log.cc:824] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:41.622609 12466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c: No bootstrap required, opened a new log
I20260808 08:19:41.627945 12466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b47afd0c64c4d399470a7fff7740b3c" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33861 } }
I20260808 08:19:41.628268 12466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:41.628396 12466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47afd0c64c4d399470a7fff7740b3c, State: Initialized, Role: FOLLOWER
I20260808 08:19:41.629354 12466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b47afd0c64c4d399470a7fff7740b3c" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33861 } }
I20260808 08:19:41.629666 12466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:19:41.629807 12466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:19:41.629994 12466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:41.631213 12466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b47afd0c64c4d399470a7fff7740b3c" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33861 } }
I20260808 08:19:41.631981 12466 leader_election.cc:302] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b47afd0c64c4d399470a7fff7740b3c; no voters:
I20260808 08:19:41.632481 12466 leader_election.cc:288] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:19:41.632644 12471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:41.632910 12471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [term 1 LEADER]: Becoming Leader. State: Replica: 8b47afd0c64c4d399470a7fff7740b3c, State: Running, Role: LEADER
I20260808 08:19:41.633457 12471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b47afd0c64c4d399470a7fff7740b3c" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33861 } }
I20260808 08:19:41.634512 12466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 08:19:41 dist-test-slave-6ntk krb5kdc[12351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:41.636608 12476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b47afd0c64c4d399470a7fff7740b3c. Latest consensus state: current_term: 1 leader_uuid: "8b47afd0c64c4d399470a7fff7740b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b47afd0c64c4d399470a7fff7740b3c" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33861 } } }
I20260808 08:19:41.636644 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b47afd0c64c4d399470a7fff7740b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b47afd0c64c4d399470a7fff7740b3c" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33861 } } }
I20260808 08:19:41.636757 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:41.636758 12476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:41.637476 12483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:19:41.643278 12483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:19:41.646710 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:41.615496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.12.13.62:33861)
0808 08:19:41.616046 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:41.616064 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:41.616516 (+ 452us) server_negotiation.cc:407] Connection header received
0808 08:19:41.622167 (+ 5651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:41.622251 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:41.622718 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:41.623045 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:41.626798 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:41.628574 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:41.630654 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:41.631155 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:41.631317 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:41.638088 (+ 6771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:41.638107 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:41.638126 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:41.638174 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:41.640697 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:41.641646 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:41.641672 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:41.641677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:41.641865 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:41.642475 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:41.642489 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:41.642491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:41.643661 (+ 1170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:41.643915 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:41.644857 (+ 942us) server_negotiation.cc:299] Negotiation successful
0808 08:19:41.645272 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":192,"threads_started":1}
I20260808 08:19:41.652117 12483 catalog_manager.cc:1347] Generated new cluster ID: b8d05129670a4cbcba4432902b26f740
I20260808 08:19:41.652209 12483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:19:41.661018 12483 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:19:41.662093 12483 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:19:41.669463 12483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b47afd0c64c4d399470a7fff7740b3c: Generated new TSK 0
I20260808 08:19:41.670655 12483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:19:41.708819 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33861
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:41.850534 12496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:41.850800 12496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:41.850845 12496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:41.854959 12496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:41.855018 12496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:41.855115 12496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:19:41.860647 12496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:41.861935 12496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:41.863068 12496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:41.873842 12504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:41.873895 12505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:41.873943 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:41.874119 12496 server_base.cc:1034] running on GCE node
I20260808 08:19:41.874670 12496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:41.875367 12496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:41.876600 12496 hybrid_clock.cc:648] HybridClock initialized: now 1786177181876584 us; error 27 us; skew 500 ppm
Aug 08 08:19:41 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177181, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:41.879544 12496 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:19:41.880465 12496 webserver.cc:460] Webserver started at http://127.12.13.1:39153/ using document root <none> and password file <none>
I20260808 08:19:41.881083 12496 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:41.881181 12496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:41.881412 12496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:41.883606 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "18e66abc53cd42d09df2244472322107"
format_stamp: "Formatted at 2026-08-08 08:19:41 on dist-test-slave-6ntk"
server_key: "bd66ba944f7d45eee60f3b940644ffb2"
server_key_iv: "87547513b7a5d7b3446264d8805a5a86"
server_key_version: "encryptionkey@0"
I20260808 08:19:41.884199 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "18e66abc53cd42d09df2244472322107"
format_stamp: "Formatted at 2026-08-08 08:19:41 on dist-test-slave-6ntk"
server_key: "bd66ba944f7d45eee60f3b940644ffb2"
server_key_iv: "87547513b7a5d7b3446264d8805a5a86"
server_key_version: "encryptionkey@0"
I20260808 08:19:41.888285 12496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 08:19:41.891080 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:41.892412 12496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:41.892565 12496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "18e66abc53cd42d09df2244472322107"
format_stamp: "Formatted at 2026-08-08 08:19:41 on dist-test-slave-6ntk"
server_key: "bd66ba944f7d45eee60f3b940644ffb2"
server_key_iv: "87547513b7a5d7b3446264d8805a5a86"
server_key_version: "encryptionkey@0"
I20260808 08:19:41.892691 12496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:41.905297 12496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:41.908851 12496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:41.909085 12496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:41.909848 12496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:41.911118 12496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:41.911201 12496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:41.911279 12496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:41.911325 12496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:41.922853 12496 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:42337
I20260808 08:19:41.922874 12672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:42337 every 8 connection(s)
I20260808 08:19:41.923888 12496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:19:41.927480 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12496
I20260808 08:19:41.927600 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:19:41.927896 12340 external_mini_cluster.cc:1428] Setting key 974c90be65576fc4cc2511be2c6ed598
Aug 08 08:19:41 dist-test-slave-6ntk krb5kdc[12351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177181, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:19:41.940933 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:41.926482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:50247 (local address 127.12.13.62:33861)
0808 08:19:41.926635 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:41.926639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:41.927729 (+ 1090us) server_negotiation.cc:407] Connection header received
0808 08:19:41.929453 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:41.929456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:41.929511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:41.929583 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:41.930949 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:41.931494 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:41.933337 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:41.933591 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:41.933690 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:41.937012 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:41.937015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:41.937017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:41.937045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:41.938631 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:41.939193 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:41.939197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:41.939199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:41.939287 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:41.939659 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:41.939663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:41.939665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:41.939915 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:41.940028 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:41.940596 (+ 568us) server_negotiation.cc:299] Negotiation successful
0808 08:19:41.940740 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:19:41.941825 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:41.926919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33861 (local address 127.12.13.1:50247)
0808 08:19:41.927531 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:41.927614 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:41.929230 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:41.929771 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:41.929779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:41.930232 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:41.930781 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:41.930796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:41.931645 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:41.931652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:41.933190 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:41.933206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:41.933770 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:41.933777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:41.934060 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:41.934799 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:41.934835 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:41.936763 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:41.938765 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:41.938772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:41.938790 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:41.939078 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:41.939394 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:41.939397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:41.939399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:41.939554 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:41.940028 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:41.940037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:41.940336 (+ 299us) client_negotiation.cc:769] Sending connection context
0808 08:19:41.940553 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 08:19:41.940916 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":225,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:19:41.943454 12673 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33861
I20260808 08:19:41.943738 12673 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:41.944419 12673 heartbeater.cc:511] Master 127.12.13.62:33861 requested a full tablet report, sending...
I20260808 08:19:41.946852 12418 ts_manager.cc:194] Registered new tserver with Master: 18e66abc53cd42d09df2244472322107 (127.12.13.1:42337)
I20260808 08:19:41.948354 12418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:50247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:19:41.962940 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33861
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:42.083422 12680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:42.083767 12680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:42.083832 12680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:42.087984 12680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:42.088058 12680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:42.088121 12680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:19:42.093602 12680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:42.094861 12680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:42.096026 12680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:42.106637 12691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:42.107026 12680 server_base.cc:1034] running on GCE node
W20260808 08:19:42.106637 12693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:42.106698 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:42.107700 12680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:42.108517 12680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:42.109714 12680 hybrid_clock.cc:648] HybridClock initialized: now 1786177182109687 us; error 40 us; skew 500 ppm
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:42.112713 12680 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:19:42.113757 12680 webserver.cc:460] Webserver started at http://127.12.13.2:43689/ using document root <none> and password file <none>
I20260808 08:19:42.114435 12680 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:42.114502 12680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:42.114678 12680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:42.116961 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "5ca15719a1e04cb0a063ef34f95acaa5"
format_stamp: "Formatted at 2026-08-08 08:19:42 on dist-test-slave-6ntk"
server_key: "fc596b9909b4d53668fa75cb837f86c7"
server_key_iv: "379ffa51109f2935f1c4e7efd9efbc83"
server_key_version: "encryptionkey@0"
I20260808 08:19:42.117566 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "5ca15719a1e04cb0a063ef34f95acaa5"
format_stamp: "Formatted at 2026-08-08 08:19:42 on dist-test-slave-6ntk"
server_key: "fc596b9909b4d53668fa75cb837f86c7"
server_key_iv: "379ffa51109f2935f1c4e7efd9efbc83"
server_key_version: "encryptionkey@0"
I20260808 08:19:42.122023 12680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260808 08:19:42.124650 12701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:42.126125 12680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:42.126266 12680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "5ca15719a1e04cb0a063ef34f95acaa5"
format_stamp: "Formatted at 2026-08-08 08:19:42 on dist-test-slave-6ntk"
server_key: "fc596b9909b4d53668fa75cb837f86c7"
server_key_iv: "379ffa51109f2935f1c4e7efd9efbc83"
server_key_version: "encryptionkey@0"
I20260808 08:19:42.126394 12680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:42.150494 12680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:42.154362 12680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:42.154613 12680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:42.155359 12680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:42.156307 12680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:42.156359 12680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:42.156431 12680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:42.156469 12680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:42.167259 12680 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:34607
I20260808 08:19:42.167282 12867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:34607 every 8 connection(s)
I20260808 08:19:42.168248 12680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:19:42.171296 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12680
I20260808 08:19:42.171412 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:19:42.171730 12340 external_mini_cluster.cc:1428] Setting key d67341b3239eff1c42d05fe1a955aced
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:19:42.185010 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:42.170350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:41323 (local address 127.12.13.62:33861)
0808 08:19:42.170486 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:42.170490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:42.171403 (+ 913us) server_negotiation.cc:407] Connection header received
0808 08:19:42.172995 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:42.172998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:42.173042 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:42.173154 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:42.174504 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.175348 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.177204 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.177634 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.177793 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:42.180642 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:42.180645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:42.180648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:42.180684 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:42.182468 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.183145 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.183148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.183149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.183299 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.183738 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.183740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.183741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.183923 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:42.184008 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:42.184762 (+ 754us) server_negotiation.cc:299] Negotiation successful
0808 08:19:42.184881 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:19:42.186060 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:42.170746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33861 (local address 127.12.13.2:41323)
0808 08:19:42.171214 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:42.171260 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:42.172765 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:42.173388 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:42.173402 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:42.173776 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:42.174354 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:42.174369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.175500 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.175505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:42.177035 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:42.177051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.177808 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.177812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:42.178054 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:42.178685 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:42.178703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:42.180432 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:42.182603 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:42.182612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:42.182632 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:42.183025 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:42.183412 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:42.183416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:42.183418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:42.183625 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:42.184036 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:42.184043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:42.184441 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 08:19:42.184663 (+ 222us) client_negotiation.cc:240] Negotiation successful
0808 08:19:42.184950 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:19:42.188104 12869 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33861
I20260808 08:19:42.188443 12869 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:42.189250 12869 heartbeater.cc:511] Master 127.12.13.62:33861 requested a full tablet report, sending...
I20260808 08:19:42.191143 12418 ts_manager.cc:194] Registered new tserver with Master: 5ca15719a1e04cb0a063ef34f95acaa5 (127.12.13.2:34607)
I20260808 08:19:42.192052 12418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:41323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:19:42.209324 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33861
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:42.331625 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:42.331971 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:42.332052 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:42.337448 12878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:42.337513 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:42.337608 12878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:19:42.343194 12878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:42.344362 12878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:42.345559 12878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:42.356165 12893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:42.356172 12890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:42.356218 12878 server_base.cc:1034] running on GCE node
W20260808 08:19:42.356189 12887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:42.356873 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:42.357609 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:42.358839 12878 hybrid_clock.cc:648] HybridClock initialized: now 1786177182358804 us; error 39 us; skew 500 ppm
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:42.361680 12878 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:19:42.362761 12878 webserver.cc:460] Webserver started at http://127.12.13.3:44687/ using document root <none> and password file <none>
I20260808 08:19:42.363380 12878 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:42.363476 12878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:42.363722 12878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:42.366106 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "751134f44d314cabb92decdbed2cd853"
format_stamp: "Formatted at 2026-08-08 08:19:42 on dist-test-slave-6ntk"
server_key: "44906230ddc1b6c0d6b96ea13649809f"
server_key_iv: "83d8c584704c5f236caa78d7f041ddef"
server_key_version: "encryptionkey@0"
I20260808 08:19:42.366701 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "751134f44d314cabb92decdbed2cd853"
format_stamp: "Formatted at 2026-08-08 08:19:42 on dist-test-slave-6ntk"
server_key: "44906230ddc1b6c0d6b96ea13649809f"
server_key_iv: "83d8c584704c5f236caa78d7f041ddef"
server_key_version: "encryptionkey@0"
I20260808 08:19:42.371050 12878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:19:42.373632 12905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:42.375130 12878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:42.375281 12878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "751134f44d314cabb92decdbed2cd853"
format_stamp: "Formatted at 2026-08-08 08:19:42 on dist-test-slave-6ntk"
server_key: "44906230ddc1b6c0d6b96ea13649809f"
server_key_iv: "83d8c584704c5f236caa78d7f041ddef"
server_key_version: "encryptionkey@0"
I20260808 08:19:42.375411 12878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:42.408715 12878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:42.412101 12878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:42.412338 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:42.413003 12878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:42.413998 12878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:42.414075 12878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:42.414151 12878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:42.414202 12878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:42.424538 12878 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:46553
I20260808 08:19:42.424559 13062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:46553 every 8 connection(s)
I20260808 08:19:42.425551 12878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:19:42.428115 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12878
I20260808 08:19:42.428275 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:19:42.428591 12340 external_mini_cluster.cc:1428] Setting key 6eba481af7eb9ceafc93448b1c63aab5
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:42.440220 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:42.427722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:57783 (local address 127.12.13.62:33861)
0808 08:19:42.427901 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:42.427906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:42.429204 (+ 1298us) server_negotiation.cc:407] Connection header received
0808 08:19:42.430445 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:42.430448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:42.430524 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:42.430671 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:42.432002 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.432548 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.433704 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.433984 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.434068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:42.436673 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:42.436675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:42.436677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:42.436704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:42.438369 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.438845 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.438848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.438849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.438926 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.439221 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.439223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.439224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.439427 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:42.439508 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:42.439990 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 08:19:42.440093 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:19:42.441104 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:42.428113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33861 (local address 127.12.13.3:57783)
0808 08:19:42.429029 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:42.429087 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:42.430176 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:42.430845 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:42.430852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:42.431344 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:42.431866 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:42.431880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.432685 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.432688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:42.433589 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:42.433599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.434118 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.434121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:42.434268 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:42.434909 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:42.434925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:42.436513 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:42.438484 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:42.438489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:42.438502 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:42.438758 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:42.439004 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:42.439006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:42.439007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:42.439141 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:42.439517 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:42.439522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:42.439757 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 08:19:42.439940 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 08:19:42.440196 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":171,"threads_started":1}
I20260808 08:19:42.442610 13065 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33861
I20260808 08:19:42.442955 13065 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:42.443603 13065 heartbeater.cc:511] Master 127.12.13.62:33861 requested a full tablet report, sending...
I20260808 08:19:42.444788 12418 ts_manager.cc:194] Registered new tserver with Master: 751134f44d314cabb92decdbed2cd853 (127.12.13.3:46553)
I20260808 08:19:42.445559 12418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:57783
I20260808 08:19:42.456063 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:19:42.487013 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:42.475947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.12.13.1:42337)
0808 08:19:42.476301 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:42.476309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:42.476337 (+ 28us) server_negotiation.cc:407] Connection header received
0808 08:19:42.476400 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:42.476405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:42.476601 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:42.476735 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:42.477510 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.478493 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.479599 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.480060 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.480159 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:42.482995 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:42.483006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:42.483022 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:42.483051 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:42.484882 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.485389 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.485395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.485396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.485486 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.485790 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.485794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.485796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.486188 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:42.486448 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:42.486638 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 08:19:42.486813 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":123,"threads_started":1}
W20260808 08:19:42.488229 12552 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56712
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[12351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:19:42.517678 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:42.508992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.12.13.1:42337)
0808 08:19:42.509171 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:42.509176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:42.509308 (+ 132us) server_negotiation.cc:407] Connection header received
0808 08:19:42.509535 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:42.509539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:42.509593 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:42.509686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:42.510532 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.511042 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.511903 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:42.512130 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:42.512247 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:42.514778 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:42.514781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:42.514783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:42.514807 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:42.516097 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.516573 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.516576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.516577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.516654 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:42.516900 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:42.516902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:42.516903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:42.517067 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:42.517173 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:42.517376 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 08:19:42.517478 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 08:19:42.519037 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12496
I20260808 08:19:42.533816 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12680
I20260808 08:19:42.552297 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12878
I20260808 08:19:42.568089 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12370
2026-08-08T08:19:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2675 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[13088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[13088](info): set up 2 sockets
Aug 08 08:19:42 dist-test-slave-6ntk krb5kdc[13088](info): commencing operation
krb5kdc: starting...
W20260808 08:19:43.832517 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:19:43 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:19:43.977707 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42785
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:42785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:44.099102 13110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:44.099457 13110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:44.099542 13110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:44.103734 13110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:19:44.103835 13110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:44.103871 13110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:44.103910 13110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:19:44.103943 13110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:19:44.109822 13110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:42785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:44.111064 13110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:44.112200 13110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:44.121594 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.121639 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.121629 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:44.121999 13110 server_base.cc:1034] running on GCE node
I20260808 08:19:44.122658 13110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:44.123868 13110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:44.125084 13110 hybrid_clock.cc:648] HybridClock initialized: now 1786177184125050 us; error 52 us; skew 500 ppm
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:44.128185 13110 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:19:44.129212 13110 webserver.cc:460] Webserver started at http://127.12.13.62:46601/ using document root <none> and password file <none>
I20260808 08:19:44.129848 13110 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:44.130007 13110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:44.130267 13110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:44.132447 13110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "eac08ff0410e4cf8ae287e8456697c23"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "bfc8ecc2586d8d4f6c2b55ccbdf48d3e"
server_key_iv: "c11d7b69f83cd5326d7b6a5aa8635e92"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.133142 13110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "eac08ff0410e4cf8ae287e8456697c23"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "bfc8ecc2586d8d4f6c2b55ccbdf48d3e"
server_key_iv: "c11d7b69f83cd5326d7b6a5aa8635e92"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.137604 13110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:19:44.140255 13133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.141701 13110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:19:44.141925 13110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "eac08ff0410e4cf8ae287e8456697c23"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "bfc8ecc2586d8d4f6c2b55ccbdf48d3e"
server_key_iv: "c11d7b69f83cd5326d7b6a5aa8635e92"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.142069 13110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:44.176517 13110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:44.179874 13110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:44.180115 13110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:44.188515 13110 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:42785
I20260808 08:19:44.188733 13211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:42785 every 8 connection(s)
I20260808 08:19:44.189620 13110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:19:44.193656 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:44.195924 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13110
I20260808 08:19:44.196058 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:19:44.196368 12340 external_mini_cluster.cc:1428] Setting key 95e2c6e87247a76546017fe697dea714
I20260808 08:19:44.201627 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23: Bootstrap starting.
I20260808 08:19:44.204497 13213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:44.205459 13213 log.cc:824] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:44.208010 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23: No bootstrap required, opened a new log
I20260808 08:19:44.213105 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:44.198039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.12.13.62:42785)
0808 08:19:44.198546 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:44.198568 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:44.198613 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:19:44.199559 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:44.199589 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:44.200058 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:44.200456 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:44.201135 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.202646 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.203541 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.204229 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.204422 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:44.207350 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:44.207357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:44.207376 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:44.207410 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:44.209462 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:44.210016 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:44.210022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:44.210029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:44.210179 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:44.210465 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:44.210468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:44.210470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:44.211246 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:44.211549 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:44.211716 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 08:19:44.212054 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":125,"threads_started":1}
I20260808 08:19:44.213645 13213 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac08ff0410e4cf8ae287e8456697c23" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42785 } }
I20260808 08:19:44.213964 13213 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:44.214026 13213 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac08ff0410e4cf8ae287e8456697c23, State: Initialized, Role: FOLLOWER
I20260808 08:19:44.214497 13213 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac08ff0410e4cf8ae287e8456697c23" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42785 } }
I20260808 08:19:44.214636 13213 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:19:44.214742 13213 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:19:44.214851 13213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:44.216053 13213 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac08ff0410e4cf8ae287e8456697c23" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42785 } }
I20260808 08:19:44.216429 13213 leader_election.cc:302] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac08ff0410e4cf8ae287e8456697c23; no voters:
I20260808 08:19:44.216777 13213 leader_election.cc:288] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:19:44.216924 13218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:44.217182 13218 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [term 1 LEADER]: Becoming Leader. State: Replica: eac08ff0410e4cf8ae287e8456697c23, State: Running, Role: LEADER
I20260808 08:19:44.217515 13218 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac08ff0410e4cf8ae287e8456697c23" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42785 } }
I20260808 08:19:44.218158 13213 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:19:44.219386 13220 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac08ff0410e4cf8ae287e8456697c23. Latest consensus state: current_term: 1 leader_uuid: "eac08ff0410e4cf8ae287e8456697c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac08ff0410e4cf8ae287e8456697c23" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42785 } } }
I20260808 08:19:44.219544 13220 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:44.219877 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eac08ff0410e4cf8ae287e8456697c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac08ff0410e4cf8ae287e8456697c23" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42785 } } }
I20260808 08:19:44.219997 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23 [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:44.219981 13232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:19:44.223680 13232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:19:44.229573 13232 catalog_manager.cc:1347] Generated new cluster ID: 3a0b4a33234d4b01b71651d2dac23c9a
I20260808 08:19:44.229664 13232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:19:44.239037 13232 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:19:44.240315 13232 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:19:44.251160 13232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eac08ff0410e4cf8ae287e8456697c23: Generated new TSK 0
I20260808 08:19:44.252069 13232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:19:44.286774 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42785
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:44.415570 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:44.416038 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:44.416121 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:44.423134 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:44.423203 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:44.423346 13251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:19:44.430462 13251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:44.431993 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:44.433444 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:44.450084 13266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:44.450227 13251 server_base.cc:1034] running on GCE node
W20260808 08:19:44.450084 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.450109 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:44.450987 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:44.451814 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:44.453013 13251 hybrid_clock.cc:648] HybridClock initialized: now 1786177184452963 us; error 67 us; skew 500 ppm
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:44.457234 13251 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:19:44.458568 13251 webserver.cc:460] Webserver started at http://127.12.13.1:44655/ using document root <none> and password file <none>
I20260808 08:19:44.459431 13251 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:44.459524 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:44.459789 13251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:44.462972 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "4af45b70a9ea4814ace70f8644558b14"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "61c240231a3a4cd5caeb274fce8bbc1c"
server_key_iv: "102e92df36e4c1b02facc3e8fc489341"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.463816 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "4af45b70a9ea4814ace70f8644558b14"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "61c240231a3a4cd5caeb274fce8bbc1c"
server_key_iv: "102e92df36e4c1b02facc3e8fc489341"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.469048 13251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 08:19:44.471999 13277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.473452 13251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 08:19:44.473631 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "4af45b70a9ea4814ace70f8644558b14"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "61c240231a3a4cd5caeb274fce8bbc1c"
server_key_iv: "102e92df36e4c1b02facc3e8fc489341"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.473747 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:44.502802 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:44.506295 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:44.506560 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:44.507246 13251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:44.508181 13251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:44.508262 13251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.508349 13251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:44.508395 13251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.518153 13251 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:45837
I20260808 08:19:44.518185 13436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:45837 every 8 connection(s)
I20260808 08:19:44.519274 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:19:44.526965 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13251
I20260808 08:19:44.527097 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:19:44.527477 12340 external_mini_cluster.cc:1428] Setting key 4be86a09301066ffe0c10d65e4a19636
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:44.534227 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:44.521350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:41563 (local address 127.12.13.62:42785)
0808 08:19:44.521496 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:44.521500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:44.522501 (+ 1001us) server_negotiation.cc:407] Connection header received
0808 08:19:44.523375 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:44.523378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:44.523429 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:44.523509 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:44.524761 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.525295 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.526532 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.526760 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.526851 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:44.529826 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:44.529851 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:44.529854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:44.529925 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:44.531791 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:44.532343 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:44.532348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:44.532350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:44.532453 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:44.532815 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:44.532818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:44.532820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:44.533061 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:44.533145 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:44.533979 (+ 834us) server_negotiation.cc:299] Negotiation successful
0808 08:19:44.534093 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:19:44.535570 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:44.521754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42785 (local address 127.12.13.1:41563)
0808 08:19:44.522316 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:44.522371 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:44.523154 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:44.523686 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:44.523693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:44.524066 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:44.524617 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:44.524631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.525465 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.525468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:44.526411 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:44.526421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.526962 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.526967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:44.527184 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:44.527884 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:44.527910 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:44.529634 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:44.531942 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:44.531948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:44.531962 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:44.532234 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:44.532566 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:44.532569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:44.532570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:44.532714 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:44.533183 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:44.533193 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:44.533625 (+ 432us) client_negotiation.cc:769] Sending connection context
0808 08:19:44.533882 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 08:19:44.534285 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":122,"threads_started":1}
I20260808 08:19:44.537042 13437 heartbeater.cc:347] Connected to a master server at 127.12.13.62:42785
I20260808 08:19:44.537350 13437 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:44.538086 13437 heartbeater.cc:511] Master 127.12.13.62:42785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:19:44.539968 13160 ts_manager.cc:194] Registered new tserver with Master: 4af45b70a9ea4814ace70f8644558b14 (127.12.13.1:45837)
I20260808 08:19:44.541256 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:41563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:19:44.562285 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42785
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:44.683023 13443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:44.683349 13443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:44.683429 13443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:44.687546 13443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:44.687650 13443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:44.687752 13443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:19:44.693245 13443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:44.694516 13443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:44.695647 13443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:44.706338 13449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.706315 13453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.706915 13448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:44.707414 13443 server_base.cc:1034] running on GCE node
I20260808 08:19:44.707894 13443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:44.708551 13443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:44.709730 13443 hybrid_clock.cc:648] HybridClock initialized: now 1786177184709714 us; error 30 us; skew 500 ppm
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:44.712780 13443 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:19:44.713675 13443 webserver.cc:460] Webserver started at http://127.12.13.2:37107/ using document root <none> and password file <none>
I20260808 08:19:44.714326 13443 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:44.714393 13443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:44.714570 13443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:44.716744 13443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "9734c09a520748ca8ca454e2f47a9784"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "517b1019f5970a638b96f2d5c137fa02"
server_key_iv: "e052da605d50ab4e03c19daba3f77684"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.717296 13443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "9734c09a520748ca8ca454e2f47a9784"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "517b1019f5970a638b96f2d5c137fa02"
server_key_iv: "e052da605d50ab4e03c19daba3f77684"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.721957 13443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260808 08:19:44.724575 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.725993 13443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:44.726121 13443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "9734c09a520748ca8ca454e2f47a9784"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "517b1019f5970a638b96f2d5c137fa02"
server_key_iv: "e052da605d50ab4e03c19daba3f77684"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.726233 13443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:44.743681 13443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:44.747387 13443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:44.747611 13443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:44.748327 13443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:44.749351 13443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:44.749401 13443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.749444 13443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:44.749459 13443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.759611 13443 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:36131
I20260808 08:19:44.759670 13617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:36131 every 8 connection(s)
I20260808 08:19:44.760636 13443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:19:44.770920 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13443
I20260808 08:19:44.771019 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:44.771328 12340 external_mini_cluster.cc:1428] Setting key 7b513a33dfbd2049a1bcd8ffeb1dd028
I20260808 08:19:44.775889 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:44.762827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:33717 (local address 127.12.13.62:42785)
0808 08:19:44.762999 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:44.763005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:44.764023 (+ 1018us) server_negotiation.cc:407] Connection header received
0808 08:19:44.765310 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:44.765313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:44.765370 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:44.765453 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:44.766900 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.767438 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.768628 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.768842 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.768922 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:44.771901 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:44.771904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:44.771906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:44.771933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:44.773530 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:44.774176 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:44.774180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:44.774182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:44.774269 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:44.774655 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:44.774659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:44.774660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:44.774871 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:44.774989 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:44.775568 (+ 579us) server_negotiation.cc:299] Negotiation successful
0808 08:19:44.775704 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 08:19:44.776786 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:44.763303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42785 (local address 127.12.13.2:33717)
0808 08:19:44.763815 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:44.763891 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:44.765087 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:44.765663 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:44.765694 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:44.766198 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:44.766745 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:44.766760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.767585 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.767588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:44.768507 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:44.768517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:44.768988 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:44.768990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:44.769134 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:44.769784 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:44.769800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:44.771684 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:44.773661 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:44.773668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:44.773685 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:44.774063 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:44.774382 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:44.774385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:44.774387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:44.774544 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:44.774988 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:44.774997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:44.775288 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 08:19:44.775508 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 08:19:44.775881 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":115,"threads_started":1}
I20260808 08:19:44.778121 13618 heartbeater.cc:347] Connected to a master server at 127.12.13.62:42785
I20260808 08:19:44.778438 13618 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:44.779162 13618 heartbeater.cc:511] Master 127.12.13.62:42785 requested a full tablet report, sending...
I20260808 08:19:44.780484 13160 ts_manager.cc:194] Registered new tserver with Master: 9734c09a520748ca8ca454e2f47a9784 (127.12.13.2:36131)
I20260808 08:19:44.781118 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:33717
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:19:44.806290 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42785
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:44.929214 13625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:44.929589 13625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:44.929646 13625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:44.934046 13625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:44.934108 13625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:44.934176 13625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:19:44.939654 13625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:44.940801 13625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:44.941839 13625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:44.953083 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.953123 13636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:44.953147 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:44.953490 13625 server_base.cc:1034] running on GCE node
I20260808 08:19:44.954079 13625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:44.954885 13625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:44.956115 13625 hybrid_clock.cc:648] HybridClock initialized: now 1786177184956061 us; error 65 us; skew 500 ppm
Aug 08 08:19:44 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:44.959386 13625 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:19:44.960295 13625 webserver.cc:460] Webserver started at http://127.12.13.3:46583/ using document root <none> and password file <none>
I20260808 08:19:44.960870 13625 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:44.960929 13625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:44.961109 13625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:44.963347 13625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "dd9913baf1cb4f29b0193760e58cfbd6"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "e986123b8bc708666eaf4e624e92f324"
server_key_iv: "7c128946cfab32a6889f6287b1742752"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.964021 13625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "dd9913baf1cb4f29b0193760e58cfbd6"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "e986123b8bc708666eaf4e624e92f324"
server_key_iv: "7c128946cfab32a6889f6287b1742752"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.968091 13625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:19:44.970698 13645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:44.972007 13625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:44.972141 13625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "dd9913baf1cb4f29b0193760e58cfbd6"
format_stamp: "Formatted at 2026-08-08 08:19:44 on dist-test-slave-6ntk"
server_key: "e986123b8bc708666eaf4e624e92f324"
server_key_iv: "7c128946cfab32a6889f6287b1742752"
server_key_version: "encryptionkey@0"
I20260808 08:19:44.972252 13625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:45.003801 13625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:45.007198 13625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:45.007433 13625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:45.008097 13625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:45.009047 13625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:45.009102 13625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:45.009177 13625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:45.009210 13625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:45.019330 13625 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:45977
I20260808 08:19:45.019349 13817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:45977 every 8 connection(s)
I20260808 08:19:45.020386 13625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:19:45.025115 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13625
I20260808 08:19:45.025264 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:19:45.025552 12340 external_mini_cluster.cc:1428] Setting key c3ac3811a1ed224c4485644864b8d90e
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:45.035177 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.022458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:50215 (local address 127.12.13.62:42785)
0808 08:19:45.022618 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.022622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.023530 (+ 908us) server_negotiation.cc:407] Connection header received
0808 08:19:45.024766 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.024768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.024852 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.024957 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:45.026554 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.027261 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.028536 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.028755 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.028844 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.031594 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:45.031597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:45.031599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:45.031626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:45.033216 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.033692 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.033695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.033696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.033776 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.034158 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.034161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.034162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.034365 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:45.034449 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.034945 (+ 496us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.035049 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:19:45.036048 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.022869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42785 (local address 127.12.13.3:50215)
0808 08:19:45.023350 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:45.023400 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:45.024554 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:45.025125 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:45.025131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:45.025584 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:45.026389 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.026410 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.027398 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.027400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.028422 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.028432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.028911 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.028914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.029057 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.029681 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:45.029708 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:45.031426 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:45.033331 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:45.033336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:45.033349 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:45.033599 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:45.033865 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:45.033920 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:45.033922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:45.034069 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:45.034463 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:45.034468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:45.034707 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 08:19:45.034878 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 08:19:45.035138 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260808 08:19:45.037235 13819 heartbeater.cc:347] Connected to a master server at 127.12.13.62:42785
I20260808 08:19:45.037523 13819 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:45.038141 13819 heartbeater.cc:511] Master 127.12.13.62:42785 requested a full tablet report, sending...
I20260808 08:19:45.039376 13160 ts_manager.cc:194] Registered new tserver with Master: dd9913baf1cb4f29b0193760e58cfbd6 (127.12.13.3:45977)
I20260808 08:19:45.040030 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:19:45.040114 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:50215
I20260808 08:19:45.052752 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.044608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.12.13.62:42785)
0808 08:19:45.044767 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.044772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.045012 (+ 240us) server_negotiation.cc:407] Connection header received
0808 08:19:45.045108 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.045111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.045168 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.045238 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:45.046058 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.046812 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.047664 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.047895 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.047979 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.049419 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:45.049422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:45.049425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:45.049449 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:45.050946 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.051361 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.051363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.051364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.051442 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.051806 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.051809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.051811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.052054 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:45.052208 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.052462 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.052578 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 08:19:45.059531 13160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:19:45.063119 13160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:45.081498 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.075422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:36131 (local address 127.0.0.1:48320)
0808 08:19:45.076522 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:45.076535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:45.076755 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:45.077302 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:45.077305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:45.077325 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:45.077506 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.077519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.078946 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.078950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.079986 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.079997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.080955 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.080957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.081047 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.081082 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 08:19:45.081177 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 08:19:45.081275 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":30,"thread_start_us":131,"threads_started":1}
I20260808 08:19:45.081532 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.075991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.12.13.2:36131)
0808 08:19:45.076419 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.076431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.076788 (+ 357us) server_negotiation.cc:407] Connection header received
0808 08:19:45.076954 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.076959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.077136 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.077336 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:45.077671 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.078778 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.080127 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.080820 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.080940 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.081100 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.081210 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.081305 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":125,"threads_started":1}
I20260808 08:19:45.082536 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.075394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.12.13.3:45977)
0808 08:19:45.076326 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.076352 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.076381 (+ 29us) server_negotiation.cc:407] Connection header received
0808 08:19:45.076628 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.076636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.076870 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.077028 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:45.077672 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.078938 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.080628 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.081493 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.082019 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.082178 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.082274 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.082383 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":188,"threads_started":1}
I20260808 08:19:45.082912 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.075270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:45977 (local address 127.0.0.1:55606)
0808 08:19:45.076224 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:45.076282 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:45.076423 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:45.077211 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:45.077216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:45.077238 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:45.077534 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.077543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.079111 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.079114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.080479 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.080491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.081943 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.081946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.082034 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.082056 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:19:45.082685 (+ 629us) client_negotiation.cc:240] Negotiation successful
0808 08:19:45.082772 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":210,"thread_start_us":112,"threads_started":1}
I20260808 08:19:45.083801 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.076056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:45837 (local address 127.0.0.1:48732)
0808 08:19:45.076580 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:45.076593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:45.076667 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:45.077409 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:45.077411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:45.077427 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:45.077607 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.077616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.079111 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.079114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.080479 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.080491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.083330 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.083334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.083452 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.083478 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 08:19:45.083568 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 08:19:45.083660 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":114,"thread_start_us":110,"threads_started":1}
I20260808 08:19:45.083890 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.076513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.12.13.1:45837)
0808 08:19:45.076945 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.076950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.076968 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:19:45.077066 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.077071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.077269 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.077416 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:45.077727 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.078981 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.082259 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.083207 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.083293 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.083395 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.083643 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.083745 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
I20260808 08:19:45.085613 13533 tablet_service.cc:1467] Processing CreateTablet for tablet 9194bff62dfa4a92b91d52081f5168e7 (DEFAULT_TABLE table=test-table [id=4c33d427ead7438c947d4b1d65e9abf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:45.086148 13713 tablet_service.cc:1467] Processing CreateTablet for tablet 9194bff62dfa4a92b91d52081f5168e7 (DEFAULT_TABLE table=test-table [id=4c33d427ead7438c947d4b1d65e9abf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:45.087148 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9194bff62dfa4a92b91d52081f5168e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:45.087229 13713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9194bff62dfa4a92b91d52081f5168e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:45.093401 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 9194bff62dfa4a92b91d52081f5168e7 (DEFAULT_TABLE table=test-table [id=4c33d427ead7438c947d4b1d65e9abf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:45.094367 13840 tablet_bootstrap.cc:492] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: Bootstrap starting.
I20260808 08:19:45.094966 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9194bff62dfa4a92b91d52081f5168e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:45.095765 13841 tablet_bootstrap.cc:492] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: Bootstrap starting.
I20260808 08:19:45.097682 13840 tablet_bootstrap.cc:654] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:45.098558 13841 tablet_bootstrap.cc:654] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:45.098675 13840 log.cc:824] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:45.099440 13841 log.cc:824] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:45.100809 13840 tablet_bootstrap.cc:492] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: No bootstrap required, opened a new log
I20260808 08:19:45.101061 13840 ts_tablet_manager.cc:1397] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260808 08:19:45.101588 13841 tablet_bootstrap.cc:492] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: No bootstrap required, opened a new log
I20260808 08:19:45.101789 13841 ts_tablet_manager.cc:1397] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:19:45.103295 13846 tablet_bootstrap.cc:492] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: Bootstrap starting.
I20260808 08:19:45.105270 13840 raft_consensus.cc:348] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.105580 13840 raft_consensus.cc:374] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:45.105656 13840 raft_consensus.cc:729] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9913baf1cb4f29b0193760e58cfbd6, State: Initialized, Role: FOLLOWER
I20260808 08:19:45.105918 13846 tablet_bootstrap.cc:654] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:45.105744 13841 raft_consensus.cc:348] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.106139 13841 raft_consensus.cc:374] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:45.106213 13841 raft_consensus.cc:729] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9734c09a520748ca8ca454e2f47a9784, State: Initialized, Role: FOLLOWER
I20260808 08:19:45.106241 13840 consensus_queue.cc:260] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.106750 13841 consensus_queue.cc:260] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.107210 13819 heartbeater.cc:503] Master 127.12.13.62:42785 was elected leader, sending a full tablet report...
I20260808 08:19:45.107391 13840 ts_tablet_manager.cc:1428] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 08:19:45.107751 13841 ts_tablet_manager.cc:1428] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:19:45.108531 13618 heartbeater.cc:503] Master 127.12.13.62:42785 was elected leader, sending a full tablet report...
I20260808 08:19:45.106993 13846 log.cc:824] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:45.112087 13846 tablet_bootstrap.cc:492] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: No bootstrap required, opened a new log
I20260808 08:19:45.112365 13846 ts_tablet_manager.cc:1397] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260808 08:19:45.116438 13846 raft_consensus.cc:348] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.116671 13846 raft_consensus.cc:374] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:45.116714 13846 raft_consensus.cc:729] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af45b70a9ea4814ace70f8644558b14, State: Initialized, Role: FOLLOWER
I20260808 08:19:45.117149 13846 consensus_queue.cc:260] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.117846 13437 heartbeater.cc:503] Master 127.12.13.62:42785 was elected leader, sending a full tablet report...
I20260808 08:19:45.118218 13846 ts_tablet_manager.cc:1428] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260808 08:19:45.228308 13848 raft_consensus.cc:482] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:45.228554 13848 raft_consensus.cc:504] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.230242 13848 leader_election.cc:288] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd9913baf1cb4f29b0193760e58cfbd6 (127.12.13.3:45977), 4af45b70a9ea4814ace70f8644558b14 (127.12.13.1:45837)
I20260808 08:19:45.235380 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.231257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:50843 (local address 127.12.13.1:45837)
0808 08:19:45.231404 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.231408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.231419 (+ 11us) server_negotiation.cc:407] Connection header received
0808 08:19:45.231491 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.231494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.231541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.231622 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:45.231885 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.232557 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.234025 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.234664 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.234778 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.234848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.235165 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.235241 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 08:19:45.235414 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.230711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:43155 (local address 127.12.13.3:45977)
0808 08:19:45.230891 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.230896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.230915 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:19:45.230978 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.230981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.231036 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.231128 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:45.231527 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.232317 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.234040 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.234664 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.234778 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.234848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.235165 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.235241 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 08:19:45.235409 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.230564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:45977 (local address 127.12.13.2:43155)
0808 08:19:45.230723 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:45.230741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:45.230884 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:45.231169 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:45.231173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:45.231200 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:45.231408 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.231415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.232457 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.232460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.233858 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.233903 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.234793 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.234796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.234999 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.235048 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 08:19:45.235129 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 08:19:45.235237 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":114,"spinlock_wait_cycles":3840}
I20260808 08:19:45.235419 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.230838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:45837 (local address 127.12.13.2:50843)
0808 08:19:45.231138 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:45.231152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:45.231240 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:45.231650 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:45.231652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:45.231670 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:45.231760 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.231764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.232673 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.232676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.233918 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:45.233925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.234793 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.234796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:45.234979 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.235049 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 08:19:45.235129 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 08:19:45.235237 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":100,"thread_start_us":105,"threads_started":1}
I20260808 08:19:45.236280 13370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9194bff62dfa4a92b91d52081f5168e7" candidate_uuid: "9734c09a520748ca8ca454e2f47a9784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af45b70a9ea4814ace70f8644558b14" is_pre_election: true
I20260808 08:19:45.236653 13370 raft_consensus.cc:2454] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9734c09a520748ca8ca454e2f47a9784 in term 0.
I20260808 08:19:45.237213 13467 leader_election.cc:302] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af45b70a9ea4814ace70f8644558b14, 9734c09a520748ca8ca454e2f47a9784; no voters:
I20260808 08:19:45.237522 13848 raft_consensus.cc:2781] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:45.237605 13848 raft_consensus.cc:482] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:45.237636 13848 raft_consensus.cc:3037] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:45.238453 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9194bff62dfa4a92b91d52081f5168e7" candidate_uuid: "9734c09a520748ca8ca454e2f47a9784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" is_pre_election: true
I20260808 08:19:45.238804 13746 raft_consensus.cc:2454] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9734c09a520748ca8ca454e2f47a9784 in term 0.
I20260808 08:19:45.238986 13848 raft_consensus.cc:504] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.239336 13848 leader_election.cc:288] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [CANDIDATE]: Term 1 election: Requested vote from peers dd9913baf1cb4f29b0193760e58cfbd6 (127.12.13.3:45977), 4af45b70a9ea4814ace70f8644558b14 (127.12.13.1:45837)
I20260808 08:19:45.239939 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9194bff62dfa4a92b91d52081f5168e7" candidate_uuid: "9734c09a520748ca8ca454e2f47a9784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9913baf1cb4f29b0193760e58cfbd6"
I20260808 08:19:45.239981 13370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9194bff62dfa4a92b91d52081f5168e7" candidate_uuid: "9734c09a520748ca8ca454e2f47a9784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af45b70a9ea4814ace70f8644558b14"
I20260808 08:19:45.240082 13746 raft_consensus.cc:3037] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:45.240144 13370 raft_consensus.cc:3037] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:45.241420 13746 raft_consensus.cc:2454] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9734c09a520748ca8ca454e2f47a9784 in term 1.
I20260808 08:19:45.241446 13370 raft_consensus.cc:2454] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9734c09a520748ca8ca454e2f47a9784 in term 1.
I20260808 08:19:45.241748 13467 leader_election.cc:302] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9734c09a520748ca8ca454e2f47a9784, dd9913baf1cb4f29b0193760e58cfbd6; no voters:
I20260808 08:19:45.241971 13848 raft_consensus.cc:2781] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:45.242262 13848 raft_consensus.cc:686] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [term 1 LEADER]: Becoming Leader. State: Replica: 9734c09a520748ca8ca454e2f47a9784, State: Running, Role: LEADER
I20260808 08:19:45.242671 13848 consensus_queue.cc:237] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } }
I20260808 08:19:45.246578 13158 catalog_manager.cc:5495] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9734c09a520748ca8ca454e2f47a9784 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "9734c09a520748ca8ca454e2f47a9784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9913baf1cb4f29b0193760e58cfbd6" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af45b70a9ea4814ace70f8644558b14" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9734c09a520748ca8ca454e2f47a9784" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 36131 } health_report { overall_health: HEALTHY } } }
W20260808 08:19:45.262102 13619 tablet.cc:2369] T 9194bff62dfa4a92b91d52081f5168e7 P 9734c09a520748ca8ca454e2f47a9784: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:19:45.270879 13438 tablet.cc:2369] T 9194bff62dfa4a92b91d52081f5168e7 P 4af45b70a9ea4814ace70f8644558b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
W20260808 08:19:45.271484 13820 tablet.cc:2369] T 9194bff62dfa4a92b91d52081f5168e7 P dd9913baf1cb4f29b0193760e58cfbd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:19:45.275422 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.265682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.12.13.1:45837)
0808 08:19:45.265924 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.265929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.266057 (+ 128us) server_negotiation.cc:407] Connection header received
0808 08:19:45.266227 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.266230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.266283 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.266374 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:45.267124 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.267658 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.268520 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.268746 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.268870 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.271811 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:45.271822 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:45.271844 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:45.271871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:45.273645 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.274135 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.274141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.274142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.274239 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.274480 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.274482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.274483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.274730 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:45.274915 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.275059 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.275174 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:19:45.307175 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.297792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.12.13.1:45837)
0808 08:19:45.298016 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.298021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.298136 (+ 115us) server_negotiation.cc:407] Connection header received
0808 08:19:45.298345 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.298348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.298405 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.298499 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:45.299294 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.299842 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.300698 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.300922 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.301030 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.303686 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:45.303690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:45.303692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:45.303720 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:45.305147 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.305630 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.305634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.305636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.305729 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.306116 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.306120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.306121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.306327 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:45.306482 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.306823 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.306972 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:19:45.338019 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:45.328620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.12.13.1:45837)
0808 08:19:45.328884 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:45.328892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:45.329038 (+ 146us) server_negotiation.cc:407] Connection header received
0808 08:19:45.329227 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:45.329230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:45.329298 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:45.329428 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:45.330389 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.331060 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.331900 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:45.332125 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:45.332212 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:45.334761 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:45.334765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:45.334767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:45.334793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:45.336129 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.336611 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.336615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.336617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.336697 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:45.336996 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:45.337000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:45.337002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:45.337200 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:45.337343 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:45.337609 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 08:19:45.337773 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260808 08:19:45.339720 13316 tablet_service.cc:2908] Rejecting scan request for tablet 9194bff62dfa4a92b91d52081f5168e7: Uninitialized: clean time has not yet been initialized
I20260808 08:19:45.342365 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13251
I20260808 08:19:45.357908 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13443
I20260808 08:19:45.372161 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13625
I20260808 08:19:45.386046 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13110
2026-08-08T08:19:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2819 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13873](info): set up 2 sockets
Aug 08 08:19:45 dist-test-slave-6ntk krb5kdc[13873](info): commencing operation
krb5kdc: starting...
W20260808 08:19:46.702185 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 08:19:46.953132 12340 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.181s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 08:19:47.058434 12340 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.105s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/test-user.keytab.
W20260808 08:19:47.534168 12340 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.476s user 0.001s sys 0.002s
Aug 08 08:19:47 dist-test-slave-6ntk krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:19:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:19:47.590152 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43405
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:47.714126 13893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:47.714423 13893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:47.714465 13893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:47.718469 13893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:19:47.718529 13893 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 08:19:47.718547 13893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:47.718565 13893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:47.718581 13893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:19:47.718597 13893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:19:47.723992 13893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:43405
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:47.725090 13893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:47.726209 13893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:19:47.735114 13893 server_base.cc:1034] running on GCE node
W20260808 08:19:47.735124 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:47.735378 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:47.735129 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:47.736001 13893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:47.737146 13893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:47.738391 13893 hybrid_clock.cc:648] HybridClock initialized: now 1786177187738369 us; error 43 us; skew 500 ppm
Aug 08 08:19:47 dist-test-slave-6ntk krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177187, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:47.741407 13893 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:19:47.742408 13893 webserver.cc:460] Webserver started at http://127.12.13.62:38621/ using document root <none> and password file <none>
I20260808 08:19:47.742975 13893 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:47.743039 13893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:47.743213 13893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:47.745325 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "c4c99474d9ae44daa6524afcd145fabb"
format_stamp: "Formatted at 2026-08-08 08:19:47 on dist-test-slave-6ntk"
server_key: "e9c7b483e6a2d849b92291b772580b0e"
server_key_iv: "0616996398b599feabcba1cd0d7223b1"
server_key_version: "encryptionkey@0"
I20260808 08:19:47.745981 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "c4c99474d9ae44daa6524afcd145fabb"
format_stamp: "Formatted at 2026-08-08 08:19:47 on dist-test-slave-6ntk"
server_key: "e9c7b483e6a2d849b92291b772580b0e"
server_key_iv: "0616996398b599feabcba1cd0d7223b1"
server_key_version: "encryptionkey@0"
I20260808 08:19:47.750394 13893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:19:47.752959 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:47.754454 13893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:47.754571 13893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "c4c99474d9ae44daa6524afcd145fabb"
format_stamp: "Formatted at 2026-08-08 08:19:47 on dist-test-slave-6ntk"
server_key: "e9c7b483e6a2d849b92291b772580b0e"
server_key_iv: "0616996398b599feabcba1cd0d7223b1"
server_key_version: "encryptionkey@0"
I20260808 08:19:47.754669 13893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:47.785792 13893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:47.789443 13893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:47.789690 13893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:47.799296 13893 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:43405
I20260808 08:19:47.799351 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:43405 every 8 connection(s)
I20260808 08:19:47.800438 13893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:19:47.803637 14007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:47.808728 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13893
I20260808 08:19:47.808842 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:19:47.809116 12340 external_mini_cluster.cc:1428] Setting key c3ed9ea9cc88f2639308bb9d58722124
I20260808 08:19:47.810005 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: Bootstrap starting.
I20260808 08:19:47.812950 14007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:47.813933 14007 log.cc:824] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:47.816237 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: No bootstrap required, opened a new log
Aug 08 08:19:47 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:47.820655 14007 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c99474d9ae44daa6524afcd145fabb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43405 } }
I20260808 08:19:47.821015 14007 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:47.821094 14007 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c99474d9ae44daa6524afcd145fabb, State: Initialized, Role: FOLLOWER
I20260808 08:19:47.821667 14007 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c99474d9ae44daa6524afcd145fabb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43405 } }
I20260808 08:19:47.821852 14007 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:19:47.821960 14007 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:19:47.822088 14007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:47.823405 14007 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c99474d9ae44daa6524afcd145fabb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43405 } }
I20260808 08:19:47.823803 14007 leader_election.cc:302] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4c99474d9ae44daa6524afcd145fabb; no voters:
I20260808 08:19:47.824237 14007 leader_election.cc:288] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:19:47.824388 14013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:47.824764 14013 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [term 1 LEADER]: Becoming Leader. State: Replica: c4c99474d9ae44daa6524afcd145fabb, State: Running, Role: LEADER
I20260808 08:19:47.825186 14013 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c99474d9ae44daa6524afcd145fabb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43405 } }
I20260808 08:19:47.825734 14007 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:19:47.826232 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:47.810756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.12.13.62:43405)
0808 08:19:47.811298 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:47.811319 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:47.811378 (+ 59us) server_negotiation.cc:407] Connection header received
0808 08:19:47.812243 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:47.812268 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:47.812712 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:47.813111 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:47.813971 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:47.815381 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:47.816263 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:47.816986 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:47.817147 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:47.819884 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:47.819892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:47.819912 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:47.819950 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:47.822121 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:47.822647 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:47.822653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:47.822659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:47.822810 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:47.823103 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:47.823107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:47.823109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:47.823716 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:47.823938 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:47.824354 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 08:19:47.824869 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":142,"threads_started":1}
I20260808 08:19:47.828243 14016 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4c99474d9ae44daa6524afcd145fabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c99474d9ae44daa6524afcd145fabb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43405 } } }
I20260808 08:19:47.828387 14016 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:47.828751 14015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4c99474d9ae44daa6524afcd145fabb. Latest consensus state: current_term: 1 leader_uuid: "c4c99474d9ae44daa6524afcd145fabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c99474d9ae44daa6524afcd145fabb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43405 } } }
I20260808 08:19:47.828840 14015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb [sys.catalog]: This master's current role is: LEADER
W20260808 08:19:47.831041 14036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:19:47.831113 14036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:19:47.831225 14037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:19:47.831965 14037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:19:47.838905 14037 catalog_manager.cc:1347] Generated new cluster ID: 838f7d45981e45aa9c9d234c28a08256
I20260808 08:19:47.839028 14037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:19:47.862260 14037 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:19:47.863339 14037 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:19:47.870520 14037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: Generated new TSK 0
I20260808 08:19:47.871352 14037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:19:47.914186 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43405
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:48.052846 14046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:48.053136 14046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:48.053180 14046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:48.057240 14046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:48.057307 14046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:48.057346 14046 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:19:48.057427 14046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:19:48.063130 14046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:48.064311 14046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:48.065439 14046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:48.075832 14056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:48.075908 14052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:48.075991 14046 server_base.cc:1034] running on GCE node
W20260808 08:19:48.076169 14053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:48.076669 14046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:48.077360 14046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:48.078593 14046 hybrid_clock.cc:648] HybridClock initialized: now 1786177188078585 us; error 33 us; skew 500 ppm
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:48.081534 14046 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:19:48.082536 14046 webserver.cc:460] Webserver started at http://127.12.13.1:33125/ using document root <none> and password file <none>
I20260808 08:19:48.083146 14046 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:48.083245 14046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:48.083508 14046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:48.085633 14046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "15c58cd846f5477a8ab695282ccb62e2"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "cf3355d6e8a3e10371cf7b03732b0edc"
server_key_iv: "0ef566639a2b11ff54ccf243a383b625"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.086316 14046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "15c58cd846f5477a8ab695282ccb62e2"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "cf3355d6e8a3e10371cf7b03732b0edc"
server_key_iv: "0ef566639a2b11ff54ccf243a383b625"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.090319 14046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:19:48.093000 14065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:48.094415 14046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:48.094580 14046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "15c58cd846f5477a8ab695282ccb62e2"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "cf3355d6e8a3e10371cf7b03732b0edc"
server_key_iv: "0ef566639a2b11ff54ccf243a383b625"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.094705 14046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:48.108588 14046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:48.111714 14046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:48.111940 14046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:48.113468 14046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:48.113551 14046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:48.113622 14046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:48.113675 14046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:48.127084 14046 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:39145
I20260808 08:19:48.127162 14225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:39145 every 8 connection(s)
I20260808 08:19:48.128257 14046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:19:48.129640 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.114483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:37705 (local address 127.12.13.62:43405)
0808 08:19:48.114673 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.114680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.115592 (+ 912us) server_negotiation.cc:407] Connection header received
0808 08:19:48.116911 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.116916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.116989 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.117059 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.119106 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.119923 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.121335 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.121590 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.121718 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.125355 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.125361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.125366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.125398 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.127262 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.127876 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.127879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.127881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.127977 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.128357 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.128364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.128365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.128585 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.128712 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.129365 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.129503 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 08:19:48.130506 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.114880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.12.13.1:37705)
0808 08:19:48.115392 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.115456 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.116611 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.117546 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.117559 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.118124 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:48.118907 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.118933 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.120103 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.120113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.121194 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.121204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.121782 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.121787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.122038 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.122764 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:48.122785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:48.125156 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:48.127411 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.127416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.127429 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.127760 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.128078 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.128082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.128084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.128238 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.128707 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:48.128714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:48.129037 (+ 323us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.129278 (+ 241us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.129587 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
I20260808 08:19:48.132210 14226 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43405
I20260808 08:19:48.132452 14226 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:48.133177 14226 heartbeater.cc:511] Master 127.12.13.62:43405 requested a full tablet report, sending...
I20260808 08:19:48.133278 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 14046
I20260808 08:19:48.133392 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:19:48.133692 12340 external_mini_cluster.cc:1428] Setting key e5197ffcc289cb295be55129590124f6
I20260808 08:19:48.135404 13939 ts_manager.cc:194] Registered new tserver with Master: 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145)
I20260808 08:19:48.136742 13939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:37705
I20260808 08:19:48.143428 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.135282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:48213 (local address 127.12.13.62:43405)
0808 08:19:48.135439 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.135443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.135459 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:19:48.135525 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.135528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.135570 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.135649 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.136262 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.137065 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.138219 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.138612 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.138750 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.139971 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.139974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.139976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.139998 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.141568 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.142173 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.142176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.142177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.142253 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.142575 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.142577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.142578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.142775 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.142854 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.143208 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.143308 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 08:19:48.143641 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.134883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.12.13.1:48213)
0808 08:19:48.135325 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.135342 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.135430 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.135770 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.135773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.135992 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:48.136118 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.136126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.137238 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.137245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.138071 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.138082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.138770 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.138773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.138863 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.139480 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:48.139492 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:48.139830 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:48.141697 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.141703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.141709 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.142062 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.142359 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.142364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.142369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.142478 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.142891 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:48.142897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:48.143027 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.143160 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.143342 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:19:48.170347 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43405
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:48.300590 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:48.300908 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:48.300956 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:48.305222 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:48.305307 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:48.305351 14236 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:19:48.305390 14236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:19:48.310957 14236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:48.312078 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:48.313128 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:48.323943 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:48.323968 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:48.323947 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:48.324180 14236 server_base.cc:1034] running on GCE node
I20260808 08:19:48.324856 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:48.325568 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:48.326756 14236 hybrid_clock.cc:648] HybridClock initialized: now 1786177188326722 us; error 42 us; skew 500 ppm
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:48.330173 14236 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:19:48.331235 14236 webserver.cc:460] Webserver started at http://127.12.13.2:41103/ using document root <none> and password file <none>
I20260808 08:19:48.331950 14236 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:48.332046 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:48.332304 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:48.334664 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "d1671e072b434dbcb441e05ea30c85d5"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "3106f33d28b2791269666cce07170109"
server_key_iv: "506b4ab639d9903024bc4f6ff4bfa632"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.335300 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "d1671e072b434dbcb441e05ea30c85d5"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "3106f33d28b2791269666cce07170109"
server_key_iv: "506b4ab639d9903024bc4f6ff4bfa632"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.339515 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:19:48.342283 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:48.343604 14236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:48.343750 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "d1671e072b434dbcb441e05ea30c85d5"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "3106f33d28b2791269666cce07170109"
server_key_iv: "506b4ab639d9903024bc4f6ff4bfa632"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.343873 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:48.358513 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:48.362542 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:48.362711 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:48.364195 14236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:48.364250 14236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:48.364293 14236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:48.364308 14236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:48.375310 14236 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:40661
I20260808 08:19:48.375525 14428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:40661 every 8 connection(s)
I20260808 08:19:48.376325 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:19:48.378906 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 14236
I20260808 08:19:48.379011 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:19:48.379340 12340 external_mini_cluster.cc:1428] Setting key 1b2cd91702985338434c46e42d3d2b23
I20260808 08:19:48.380275 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.365209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:52943 (local address 127.12.13.62:43405)
0808 08:19:48.365383 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.365391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.366289 (+ 898us) server_negotiation.cc:407] Connection header received
0808 08:19:48.367493 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.367499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.367571 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.367678 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.369572 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.370489 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.372346 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.372564 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.372649 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.376002 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.376006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.376009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.376044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.377730 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.378417 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.378420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.378422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.378516 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.378950 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.378953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.378954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.379252 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.379340 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.379961 (+ 621us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.380086 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:19:48.381282 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.365608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.12.13.2:52943)
0808 08:19:48.366117 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.366168 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.367233 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.368015 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.368024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.368588 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:48.369380 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.369404 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.370643 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.370647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.372194 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.372209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.372743 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.372749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.373003 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.373776 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:48.373804 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:48.375822 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:48.377864 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.377893 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.377908 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.378295 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.378615 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.378619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.378621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.378816 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.379346 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:48.379356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:48.379658 (+ 302us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.379897 (+ 239us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.380217 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260808 08:19:48.383049 14429 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43405
I20260808 08:19:48.383225 14429 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:48.383761 14429 heartbeater.cc:511] Master 127.12.13.62:43405 requested a full tablet report, sending...
I20260808 08:19:48.384918 13939 ts_manager.cc:194] Registered new tserver with Master: d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661)
I20260808 08:19:48.385653 13939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:52943
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:19:48.394276 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.385715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:37321 (local address 127.12.13.62:43405)
0808 08:19:48.385838 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.385842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.386173 (+ 331us) server_negotiation.cc:407] Connection header received
0808 08:19:48.386311 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.386314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.386414 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.386499 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.387146 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.387874 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.389237 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.389452 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.389535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.390904 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.390907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.390909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.390932 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.392526 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.392994 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.392997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.392998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.393076 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.393364 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.393366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.393368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.393584 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.393667 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.394034 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.394146 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:19:48.394297 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.385634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.12.13.2:37321)
0808 08:19:48.386079 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.386092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.386217 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.386555 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.386560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.386829 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:48.387000 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.387012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.388035 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.388040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.389107 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.389120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.389583 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.389588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.389701 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.390249 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:48.390266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:48.390727 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:48.392661 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.392664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.392665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.392898 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.393166 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.393168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.393170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.393253 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.393672 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:48.393674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:48.393766 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.393849 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.394122 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:19:48.413841 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43405
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:48.535727 14440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:48.536002 14440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:48.536087 14440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:48.540149 14440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:48.540216 14440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:48.540300 14440 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:19:48.540393 14440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:19:48.546048 14440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:48.547267 14440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:48.548467 14440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:48.558947 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:48.558947 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:48.558944 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:48.559301 14440 server_base.cc:1034] running on GCE node
I20260808 08:19:48.559893 14440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:48.560647 14440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:48.561841 14440 hybrid_clock.cc:648] HybridClock initialized: now 1786177188561820 us; error 45 us; skew 500 ppm
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:48.564858 14440 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:19:48.566004 14440 webserver.cc:460] Webserver started at http://127.12.13.3:33871/ using document root <none> and password file <none>
I20260808 08:19:48.566632 14440 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:48.566735 14440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:48.566975 14440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:48.569221 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "6e34c1f115c34022af6b10a30978601a"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "e6ea68edda6ee8be7faf771f05992bae"
server_key_iv: "c76e674e35fc5633557438f1677adbff"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.569928 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "6e34c1f115c34022af6b10a30978601a"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "e6ea68edda6ee8be7faf771f05992bae"
server_key_iv: "c76e674e35fc5633557438f1677adbff"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.574210 14440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:19:48.576954 14457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:48.578454 14440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:48.578621 14440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "6e34c1f115c34022af6b10a30978601a"
format_stamp: "Formatted at 2026-08-08 08:19:48 on dist-test-slave-6ntk"
server_key: "e6ea68edda6ee8be7faf771f05992bae"
server_key_iv: "c76e674e35fc5633557438f1677adbff"
server_key_version: "encryptionkey@0"
I20260808 08:19:48.578758 14440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:48.608073 14440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:48.612053 14440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:48.612308 14440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:48.614106 14440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:48.614184 14440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:48.614248 14440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:48.614283 14440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:48.626247 14440 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:43133
I20260808 08:19:48.626296 14620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:43133 every 8 connection(s)
I20260808 08:19:48.627643 14440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:19:48.630795 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.614990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:60289 (local address 127.12.13.62:43405)
0808 08:19:48.615183 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.615200 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.616190 (+ 990us) server_negotiation.cc:407] Connection header received
0808 08:19:48.617620 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.617626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.617704 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.617774 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.620044 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.620603 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.621933 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.622237 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.622391 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.625948 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.625953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.625958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.625998 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.628196 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.628859 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.628866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.628868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.628970 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.629581 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.629584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.629585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.629797 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.629954 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.630550 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.630669 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:19:48.631834 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.615421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.12.13.3:60289)
0808 08:19:48.615950 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.616033 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.617284 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.618352 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.618363 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.618946 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:48.619830 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.619856 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.620760 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.620764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.621715 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.621726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.622452 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.622457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.622644 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.623984 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:48.624002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:48.625621 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:48.628339 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.628345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.628362 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.628731 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.629324 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.629327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.629328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.629481 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.630038 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:48.630044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:48.630308 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.630497 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.630781 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":107,"threads_started":1}
I20260808 08:19:48.632510 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 14440
I20260808 08:19:48.632647 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:19:48.632982 12340 external_mini_cluster.cc:1428] Setting key ccc042c7f044c29455855d352fb30184
I20260808 08:19:48.633401 14621 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43405
I20260808 08:19:48.633615 14621 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:48.634305 14621 heartbeater.cc:511] Master 127.12.13.62:43405 requested a full tablet report, sending...
I20260808 08:19:48.635488 13935 ts_manager.cc:194] Registered new tserver with Master: 6e34c1f115c34022af6b10a30978601a (127.12.13.3:43133)
I20260808 08:19:48.636216 13935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:60289
I20260808 08:19:48.644605 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.636026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.12.13.3:38771)
0808 08:19:48.636395 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.636411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.636553 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.637980 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.637983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.638184 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:48.638295 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.638302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.639087 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.639091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.639674 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.639684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.640171 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.640175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.640268 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.640807 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:48.640843 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:48.641198 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:48.642954 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.642959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.642961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.643175 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.643483 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:48.643487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:48.643489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:48.643580 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:48.644014 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:48.644017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:48.644125 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.644238 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.644398 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260808 08:19:48.644600 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.636094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:38771 (local address 127.12.13.62:43405)
0808 08:19:48.636213 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.636217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.636482 (+ 265us) server_negotiation.cc:407] Connection header received
0808 08:19:48.636641 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.636644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.636686 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.636746 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.638430 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.638946 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.639811 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.640043 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.640163 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.641363 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.641366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.641368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.641392 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.642835 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.643284 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.643288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.643290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.643382 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.643685 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.643688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.643690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.643907 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.644026 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.644299 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.644428 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:19:48.647609 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 08:19:48 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:48.676457 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.668007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.12.13.62:43405)
0808 08:19:48.668176 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.668181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.668199 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:19:48.668258 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.668261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.668305 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.668387 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:48.669190 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.669959 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.670782 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.671005 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.671089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.673602 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:48.673605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:48.673608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:48.673634 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:48.675007 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.675407 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.675410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.675411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.675488 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:48.675723 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:48.675725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:48.675726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:48.675903 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:48.676004 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.676218 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.676328 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:19:48.679642 13935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:19:48.682816 13935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:48.698770 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.693043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.12.13.2:40661)
0808 08:19:48.693785 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.693797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.693819 (+ 22us) server_negotiation.cc:407] Connection header received
0808 08:19:48.693923 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.693931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.694209 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.694363 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:48.694624 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.695678 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.697077 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.697973 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.698095 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.698299 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.698487 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.698600 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":122,"threads_started":1}
I20260808 08:19:48.698760 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.692020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:39145 (local address 127.0.0.1:47850)
0808 08:19:48.692457 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.692518 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.692649 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.693598 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.693607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.693631 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:48.693787 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.693795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.695526 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.695531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.697137 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.697148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.698285 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.698291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.698390 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.698443 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.698501 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.698612 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":179,"thread_start_us":136,"threads_started":1}
I20260808 08:19:48.698935 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.692866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40661 (local address 127.0.0.1:49540)
0808 08:19:48.693415 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.693429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.693518 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.694380 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.694385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.694409 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:48.694499 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.694504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.695795 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.695799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.696929 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.696941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.698107 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.698111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.698242 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.698294 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.698431 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.698586 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260808 08:19:48.699734 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.692237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.12.13.1:39145)
0808 08:19:48.693057 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.693065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.693091 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:19:48.693180 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.693186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.693416 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.694000 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:48.694154 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.695353 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.697273 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.698099 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.699314 (+ 1215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.699466 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.699517 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.699606 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":152,"threads_started":1}
I20260808 08:19:48.699970 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.693197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.12.13.3:43133)
0808 08:19:48.693649 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.693661 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.693684 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:19:48.693761 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.693765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.694013 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.694200 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:48.694486 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.695258 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.697984 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.699331 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.699475 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.699647 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.699701 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.699789 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":116,"threads_started":1}
I20260808 08:19:48.700548 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.692392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:43133 (local address 127.0.0.1:48748)
0808 08:19:48.692963 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.692977 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.693064 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.694178 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.694181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.694201 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:48.694334 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.694357 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.695409 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.695414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.697137 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.697148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.699473 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.699477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.699566 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.699588 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.699659 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.699735 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":194,"thread_start_us":112,"threads_started":1}
I20260808 08:19:48.702478 14138 tablet_service.cc:1467] Processing CreateTablet for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 (DEFAULT_TABLE table=test-table [id=cb13ab08df714177b9c88fee3dca456b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:48.702370 14332 tablet_service.cc:1467] Processing CreateTablet for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 (DEFAULT_TABLE table=test-table [id=cb13ab08df714177b9c88fee3dca456b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:48.703480 14138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:48.703512 14332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:48.703433 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 (DEFAULT_TABLE table=test-table [id=cb13ab08df714177b9c88fee3dca456b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:48.704487 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:48.709932 14654 tablet_bootstrap.cc:492] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Bootstrap starting.
I20260808 08:19:48.712354 14657 tablet_bootstrap.cc:492] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Bootstrap starting.
I20260808 08:19:48.712839 14654 tablet_bootstrap.cc:654] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:48.713574 14658 tablet_bootstrap.cc:492] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Bootstrap starting.
I20260808 08:19:48.713749 14654 log.cc:824] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:48.714640 14657 tablet_bootstrap.cc:654] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:48.715458 14657 log.cc:824] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:48.715852 14658 tablet_bootstrap.cc:654] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:48.716061 14654 tablet_bootstrap.cc:492] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: No bootstrap required, opened a new log
I20260808 08:19:48.716269 14654 ts_tablet_manager.cc:1397] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 08:19:48.716696 14658 log.cc:824] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:48.717589 14657 tablet_bootstrap.cc:492] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: No bootstrap required, opened a new log
I20260808 08:19:48.717792 14657 ts_tablet_manager.cc:1397] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:19:48.718514 14658 tablet_bootstrap.cc:492] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: No bootstrap required, opened a new log
I20260808 08:19:48.718734 14658 ts_tablet_manager.cc:1397] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 08:19:48.720903 14654 raft_consensus.cc:348] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.721208 14654 raft_consensus.cc:374] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:48.721282 14654 raft_consensus.cc:729] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c58cd846f5477a8ab695282ccb62e2, State: Initialized, Role: FOLLOWER
I20260808 08:19:48.721822 14654 consensus_queue.cc:260] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.721773 14657 raft_consensus.cc:348] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.722049 14657 raft_consensus.cc:374] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:48.722110 14657 raft_consensus.cc:729] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1671e072b434dbcb441e05ea30c85d5, State: Initialized, Role: FOLLOWER
I20260808 08:19:48.721971 14658 raft_consensus.cc:348] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.722219 14658 raft_consensus.cc:374] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:48.722276 14658 raft_consensus.cc:729] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e34c1f115c34022af6b10a30978601a, State: Initialized, Role: FOLLOWER
I20260808 08:19:48.722533 14657 consensus_queue.cc:260] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.722739 14658 consensus_queue.cc:260] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.722942 14654 ts_tablet_manager.cc:1428] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 08:19:48.723469 14226 heartbeater.cc:503] Master 127.12.13.62:43405 was elected leader, sending a full tablet report...
I20260808 08:19:48.723491 14657 ts_tablet_manager.cc:1428] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 08:19:48.723621 14621 heartbeater.cc:503] Master 127.12.13.62:43405 was elected leader, sending a full tablet report...
I20260808 08:19:48.723711 14658 ts_tablet_manager.cc:1428] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:19:48.723788 14429 heartbeater.cc:503] Master 127.12.13.62:43405 was elected leader, sending a full tablet report...
W20260808 08:19:48.877599 14430 tablet.cc:2369] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:19:48.879096 14623 tablet.cc:2369] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:19:48.879710 14227 tablet.cc:2369] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:19:48.983659 14665 raft_consensus.cc:482] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:48.983964 14665 raft_consensus.cc:504] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.985196 14665 leader_election.cc:288] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145), d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661)
I20260808 08:19:48.989202 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.985573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:39145 (local address 127.12.13.3:48353)
0808 08:19:48.985753 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.985771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.985954 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.986275 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.986278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.986294 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:48.986454 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.986460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.987232 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.987236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.988025 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.988034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.988739 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.988743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.988852 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.988875 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.988980 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.989059 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 08:19:48.989202 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.985629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:48353 (local address 127.12.13.1:39145)
0808 08:19:48.985783 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.985791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.985904 (+ 113us) server_negotiation.cc:407] Connection header received
0808 08:19:48.986093 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.986099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.986164 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.986256 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:48.986582 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.987102 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.988167 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.988603 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.988736 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.988806 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.988997 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.989073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 08:19:48.989780 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.985950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40661 (local address 127.12.13.3:42119)
0808 08:19:48.986363 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:48.986375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:48.986500 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:48.986741 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:48.986744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:48.986759 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:48.987152 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.987159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.987892 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.987898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.988654 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:48.988662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.989318 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.989322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:48.989426 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.989453 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:19:48.989566 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 08:19:48.989639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":141,"threads_started":1}
I20260808 08:19:48.989799 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:48.986070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:42119 (local address 127.12.13.2:40661)
0808 08:19:48.986217 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:48.986223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:48.986465 (+ 242us) server_negotiation.cc:407] Connection header received
0808 08:19:48.986591 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:48.986594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:48.986644 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:48.986733 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:48.987281 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.987767 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.988789 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:48.989185 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:48.989315 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:48.989387 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:48.989603 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 08:19:48.989704 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:19:48.989991 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3155e5dbda7c4d81bfb147c4d9f7c3e4" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c58cd846f5477a8ab695282ccb62e2" is_pre_election: true
I20260808 08:19:48.990371 14165 raft_consensus.cc:2454] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 0.
I20260808 08:19:48.990581 14353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3155e5dbda7c4d81bfb147c4d9f7c3e4" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1671e072b434dbcb441e05ea30c85d5" is_pre_election: true
I20260808 08:19:48.990913 14353 raft_consensus.cc:2454] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 0.
I20260808 08:19:48.990998 14459 leader_election.cc:302] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c58cd846f5477a8ab695282ccb62e2, 6e34c1f115c34022af6b10a30978601a; no voters:
I20260808 08:19:48.991365 14665 raft_consensus.cc:2781] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:48.991465 14665 raft_consensus.cc:482] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:48.991510 14665 raft_consensus.cc:3037] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:48.992681 14665 raft_consensus.cc:504] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:48.992972 14665 leader_election.cc:288] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 election: Requested vote from peers 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145), d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661)
I20260808 08:19:48.993403 14353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3155e5dbda7c4d81bfb147c4d9f7c3e4" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1671e072b434dbcb441e05ea30c85d5"
I20260808 08:19:48.993403 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3155e5dbda7c4d81bfb147c4d9f7c3e4" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c58cd846f5477a8ab695282ccb62e2"
I20260808 08:19:48.993546 14353 raft_consensus.cc:3037] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:48.993556 14165 raft_consensus.cc:3037] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:48.994920 14353 raft_consensus.cc:2454] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 1.
I20260808 08:19:48.994920 14165 raft_consensus.cc:2454] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 1.
I20260808 08:19:48.995311 14459 leader_election.cc:302] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c58cd846f5477a8ab695282ccb62e2, 6e34c1f115c34022af6b10a30978601a; no voters:
I20260808 08:19:48.995535 14665 raft_consensus.cc:2781] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:48.995831 14665 raft_consensus.cc:686] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 1 LEADER]: Becoming Leader. State: Replica: 6e34c1f115c34022af6b10a30978601a, State: Running, Role: LEADER
I20260808 08:19:48.996229 14665 consensus_queue.cc:237] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.000061 13934 catalog_manager.cc:5495] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e34c1f115c34022af6b10a30978601a (127.12.13.3). New cstate: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } health_report { overall_health: HEALTHY } } }
I20260808 08:19:49.080274 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.076429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.12.13.3:43133)
0808 08:19:49.076571 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.076575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.076668 (+ 93us) server_negotiation.cc:407] Connection header received
0808 08:19:49.076857 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.076860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.076919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.077010 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:49.077273 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.077797 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.078678 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.078905 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.078992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.079340 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.079822 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.079936 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.080093 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.080154 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:19:49.087849 14353 raft_consensus.cc:1261] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Refusing update from remote peer 6e34c1f115c34022af6b10a30978601a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:49.088698 14665 consensus_queue.cc:1035] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:49.089515 14165 raft_consensus.cc:1261] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Refusing update from remote peer 6e34c1f115c34022af6b10a30978601a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:49.090170 14665 consensus_queue.cc:1035] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:49.097191 14672 mvcc.cc:205] Tried to move back new op lower bound from 7316181766493884416 to 7316181766131322880. Current Snapshot: MvccSnapshot[applied={T|T < 7316181766493884416}]
I20260808 08:19:49.098367 14673 mvcc.cc:205] Tried to move back new op lower bound from 7316181766493884416 to 7316181766131322880. Current Snapshot: MvccSnapshot[applied={T|T < 7316181766493884416}]
I20260808 08:19:49.112340 13939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:49.112628 13939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:49.115690 13939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: Sending DeleteTablet for 3 replicas of tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4
I20260808 08:19:49.116648 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:48748
I20260808 08:19:49.116674 14332 tablet_service.cc:1514] Processing DeleteTablet for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:49540
I20260808 08:19:49.116755 14138 tablet_service.cc:1514] Processing DeleteTablet for tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:47850
I20260808 08:19:49.117286 14681 tablet_replica.cc:330] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: stopping tablet replica
I20260808 08:19:49.117672 14686 tablet_replica.cc:330] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: stopping tablet replica
I20260808 08:19:49.117691 14681 raft_consensus.cc:2229] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:49.118017 14686 raft_consensus.cc:2229] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:49.118088 14681 raft_consensus.cc:2258] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:49.118317 14686 raft_consensus.cc:2258] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:49.118527 14682 tablet_replica.cc:330] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: stopping tablet replica
I20260808 08:19:49.118505 13939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 08:19:49.118799 14682 raft_consensus.cc:2229] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 1 LEADER]: Raft consensus shutting down.
W20260808 08:19:49.119042 13939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:49.119755 14682 raft_consensus.cc:2258] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:49.120416 14686 ts_tablet_manager.cc:1905] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:49.122231 14681 ts_tablet_manager.cc:1905] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:49.122421 14682 ts_tablet_manager.cc:1905] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:49.125149 14681 ts_tablet_manager.cc:1918] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:19:49.125257 14681 log.cc:1196] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/3155e5dbda7c4d81bfb147c4d9f7c3e4
I20260808 08:19:49.125255 14682 ts_tablet_manager.cc:1918] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:19:49.125351 14682 log.cc:1196] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/3155e5dbda7c4d81bfb147c4d9f7c3e4
I20260808 08:19:49.125581 14681 ts_tablet_manager.cc:1939] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P d1671e072b434dbcb441e05ea30c85d5: Deleting consensus metadata
I20260808 08:19:49.125697 14682 ts_tablet_manager.cc:1939] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 6e34c1f115c34022af6b10a30978601a: Deleting consensus metadata
I20260808 08:19:49.127153 14138 tablet_service.cc:1467] Processing CreateTablet for tablet 83d99c6af35b41faade8ff55b8e7b349 (DEFAULT_TABLE table=test-table [id=fc1468269aed4f01a7e7363ffbef997d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:49.127449 14138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d99c6af35b41faade8ff55b8e7b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:49.129419 14686 ts_tablet_manager.cc:1918] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:19:49.129511 14686 log.cc:1196] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/3155e5dbda7c4d81bfb147c4d9f7c3e4
I20260808 08:19:49.129838 14686 ts_tablet_manager.cc:1939] T 3155e5dbda7c4d81bfb147c4d9f7c3e4 P 15c58cd846f5477a8ab695282ccb62e2: Deleting consensus metadata
I20260808 08:19:49.129925 14654 tablet_bootstrap.cc:492] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Bootstrap starting.
I20260808 08:19:49.130877 14654 tablet_bootstrap.cc:654] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:49.132460 13918 catalog_manager.cc:4841] TS 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145): tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 (table test-table [id=cb13ab08df714177b9c88fee3dca456b]) successfully deleted
I20260808 08:19:49.133031 13918 catalog_manager.cc:4841] TS d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661): tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 (table test-table [id=cb13ab08df714177b9c88fee3dca456b]) successfully deleted
I20260808 08:19:49.133279 14332 tablet_service.cc:1467] Processing CreateTablet for tablet 83d99c6af35b41faade8ff55b8e7b349 (DEFAULT_TABLE table=test-table [id=fc1468269aed4f01a7e7363ffbef997d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:49.133558 14332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d99c6af35b41faade8ff55b8e7b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:49.134646 14654 tablet_bootstrap.cc:492] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: No bootstrap required, opened a new log
I20260808 08:19:49.134753 14654 ts_tablet_manager.cc:1397] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 08:19:49.135221 14654 raft_consensus.cc:348] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.135321 14654 raft_consensus.cc:374] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:49.135350 14654 raft_consensus.cc:729] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c58cd846f5477a8ab695282ccb62e2, State: Initialized, Role: FOLLOWER
I20260808 08:19:49.135625 14654 consensus_queue.cc:260] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.136161 14657 tablet_bootstrap.cc:492] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Bootstrap starting.
I20260808 08:19:49.137161 14657 tablet_bootstrap.cc:654] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:49.138761 14654 ts_tablet_manager.cc:1428] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260808 08:19:49.142139 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 83d99c6af35b41faade8ff55b8e7b349 (DEFAULT_TABLE table=test-table [id=fc1468269aed4f01a7e7363ffbef997d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:49.142527 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d99c6af35b41faade8ff55b8e7b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:49.143004 13919 catalog_manager.cc:4841] TS 6e34c1f115c34022af6b10a30978601a (127.12.13.3:43133): tablet 3155e5dbda7c4d81bfb147c4d9f7c3e4 (table test-table [id=cb13ab08df714177b9c88fee3dca456b]) successfully deleted
I20260808 08:19:49.145115 14657 tablet_bootstrap.cc:492] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: No bootstrap required, opened a new log
I20260808 08:19:49.145257 14657 ts_tablet_manager.cc:1397] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260808 08:19:49.145815 14657 raft_consensus.cc:348] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.146040 14657 raft_consensus.cc:374] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:49.146071 14658 tablet_bootstrap.cc:492] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Bootstrap starting.
I20260808 08:19:49.146102 14657 raft_consensus.cc:729] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1671e072b434dbcb441e05ea30c85d5, State: Initialized, Role: FOLLOWER
I20260808 08:19:49.146286 14657 consensus_queue.cc:260] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.146679 14657 ts_tablet_manager.cc:1428] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:49.147184 14658 tablet_bootstrap.cc:654] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:49.148655 14658 tablet_bootstrap.cc:492] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: No bootstrap required, opened a new log
I20260808 08:19:49.148792 14658 ts_tablet_manager.cc:1397] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:19:49.149197 14658 raft_consensus.cc:348] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.149351 14658 raft_consensus.cc:374] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:49.149397 14658 raft_consensus.cc:729] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e34c1f115c34022af6b10a30978601a, State: Initialized, Role: FOLLOWER
I20260808 08:19:49.149546 14658 consensus_queue.cc:260] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.149896 14658 ts_tablet_manager.cc:1428] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:49.171816 14664 raft_consensus.cc:482] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:49.171996 14664 raft_consensus.cc:504] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.173229 14664 leader_election.cc:288] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145), 6e34c1f115c34022af6b10a30978601a (127.12.13.3:43133)
I20260808 08:19:49.178182 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.173969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:47433 (local address 127.12.13.3:43133)
0808 08:19:49.174100 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.174104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.174119 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:19:49.174236 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.174241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.174309 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.174415 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:49.174707 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.175484 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.177087 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.177483 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.177616 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.177688 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.177950 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.178040 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:19:49.178400 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.173655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:42637 (local address 127.12.13.1:39145)
0808 08:19:49.173836 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.173841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.174040 (+ 199us) server_negotiation.cc:407] Connection header received
0808 08:19:49.174335 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.174339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.174391 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.174463 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:49.174804 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.175294 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.177078 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.177712 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.177827 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.177931 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.178085 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.178168 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 08:19:49.178514 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.173490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:39145 (local address 127.12.13.2:42637)
0808 08:19:49.173948 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:49.173964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:49.174145 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:49.174544 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:49.174547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:49.174561 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:49.174688 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.174695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.175425 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.175431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.176888 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.176900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.177822 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.177825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.177956 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.177978 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:19:49.178048 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 08:19:49.178115 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":47,"spinlock_wait_cycles":17152,"thread_start_us":92,"threads_started":1}
I20260808 08:19:49.179124 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.173530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:43133 (local address 127.12.13.2:47433)
0808 08:19:49.174002 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:49.174017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:49.174132 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:49.174427 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:49.174430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:49.174446 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:49.174579 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.174586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.175598 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.175601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.176888 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.176900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.177635 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.177638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.177756 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.177788 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 08:19:49.178899 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0808 08:19:49.178976 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":23,"spinlock_wait_cycles":4480,"thread_start_us":117,"threads_started":1}
I20260808 08:19:49.179219 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d99c6af35b41faade8ff55b8e7b349" candidate_uuid: "d1671e072b434dbcb441e05ea30c85d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c58cd846f5477a8ab695282ccb62e2" is_pre_election: true
I20260808 08:19:49.179383 14165 raft_consensus.cc:2454] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1671e072b434dbcb441e05ea30c85d5 in term 0.
I20260808 08:19:49.179780 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d99c6af35b41faade8ff55b8e7b349" candidate_uuid: "d1671e072b434dbcb441e05ea30c85d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e34c1f115c34022af6b10a30978601a" is_pre_election: true
I20260808 08:19:49.179838 14260 leader_election.cc:302] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c58cd846f5477a8ab695282ccb62e2, d1671e072b434dbcb441e05ea30c85d5; no voters:
I20260808 08:19:49.180027 14563 raft_consensus.cc:2454] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1671e072b434dbcb441e05ea30c85d5 in term 0.
I20260808 08:19:49.180112 14664 raft_consensus.cc:2781] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:49.180164 14664 raft_consensus.cc:482] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:49.180229 14664 raft_consensus.cc:3037] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:49.181341 14664 raft_consensus.cc:504] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.181779 14664 leader_election.cc:288] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [CANDIDATE]: Term 1 election: Requested vote from peers 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145), 6e34c1f115c34022af6b10a30978601a (127.12.13.3:43133)
I20260808 08:19:49.182158 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d99c6af35b41faade8ff55b8e7b349" candidate_uuid: "d1671e072b434dbcb441e05ea30c85d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c58cd846f5477a8ab695282ccb62e2"
I20260808 08:19:49.182178 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d99c6af35b41faade8ff55b8e7b349" candidate_uuid: "d1671e072b434dbcb441e05ea30c85d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e34c1f115c34022af6b10a30978601a"
I20260808 08:19:49.182301 14165 raft_consensus.cc:3037] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:49.182330 14563 raft_consensus.cc:3037] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:49.183526 14165 raft_consensus.cc:2454] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1671e072b434dbcb441e05ea30c85d5 in term 1.
I20260808 08:19:49.183583 14563 raft_consensus.cc:2454] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1671e072b434dbcb441e05ea30c85d5 in term 1.
I20260808 08:19:49.183919 14262 leader_election.cc:302] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e34c1f115c34022af6b10a30978601a, d1671e072b434dbcb441e05ea30c85d5; no voters:
I20260808 08:19:49.184108 14664 raft_consensus.cc:2781] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:49.184360 14664 raft_consensus.cc:686] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 1 LEADER]: Becoming Leader. State: Replica: d1671e072b434dbcb441e05ea30c85d5, State: Running, Role: LEADER
I20260808 08:19:49.184653 14664 consensus_queue.cc:237] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.187471 13939 catalog_manager.cc:5495] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "d1671e072b434dbcb441e05ea30c85d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } health_report { overall_health: UNKNOWN } } }
Aug 08 08:19:49 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177187, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:49.230290 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.219893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43405 (local address 127.0.0.1:41864)
0808 08:19:49.220168 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:49.220196 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:49.220308 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:49.220618 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:49.220620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:49.221013 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:49.221113 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.221119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.222422 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.222425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.223007 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.223016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.223695 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.223703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.223773 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.224487 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:49.224500 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:49.226299 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:49.228570 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:49.228580 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:49.228586 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:49.228842 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:49.229171 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:49.229173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:49.229175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:49.229258 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:49.229722 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:49.229728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:49.229845 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 08:19:49.230014 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 08:19:49.230144 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260808 08:19:49.230453 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.219993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.12.13.62:43405)
0808 08:19:49.220272 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.220277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.220297 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:19:49.220405 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.220409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.220462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.220575 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:49.221306 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.222243 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.223144 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.223539 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.223656 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.226481 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:49.226484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:49.226486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:49.226509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:49.228449 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:49.228954 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:49.228958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:49.228960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:49.229065 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:49.229354 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:49.229357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:49.229359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:49.229608 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:49.229715 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.230097 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.230236 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260808 08:19:49.235658 13935 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:41864:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 08:19:49.236456 13935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:49.242336 14138 tablet_service.cc:1467] Processing CreateTablet for tablet f1b506f4153d43b6b37a321c5e3b3509 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c1eb358784846f3b43c1e81e2b4df23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:19:49.242345 14332 tablet_service.cc:1467] Processing CreateTablet for tablet f1b506f4153d43b6b37a321c5e3b3509 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c1eb358784846f3b43c1e81e2b4df23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:19:49.242816 14138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b506f4153d43b6b37a321c5e3b3509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:49.242846 14332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b506f4153d43b6b37a321c5e3b3509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:49.242339 14532 tablet_service.cc:1467] Processing CreateTablet for tablet f1b506f4153d43b6b37a321c5e3b3509 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c1eb358784846f3b43c1e81e2b4df23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:19:49.244642 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b506f4153d43b6b37a321c5e3b3509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:49.245780 14654 tablet_bootstrap.cc:492] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: Bootstrap starting.
I20260808 08:19:49.246009 14657 tablet_bootstrap.cc:492] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: Bootstrap starting.
I20260808 08:19:49.246940 14657 tablet_bootstrap.cc:654] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:49.246941 14654 tablet_bootstrap.cc:654] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:49.248358 14654 tablet_bootstrap.cc:492] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: No bootstrap required, opened a new log
I20260808 08:19:49.248451 14654 ts_tablet_manager.cc:1397] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 08:19:49.248939 14654 raft_consensus.cc:348] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.249079 14654 raft_consensus.cc:374] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:49.249117 14654 raft_consensus.cc:729] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c58cd846f5477a8ab695282ccb62e2, State: Initialized, Role: FOLLOWER
I20260808 08:19:49.249249 14654 consensus_queue.cc:260] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.249629 14663 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.249796 14663 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.249835 14658 tablet_bootstrap.cc:492] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: Bootstrap starting.
I20260808 08:19:49.250270 14654 ts_tablet_manager.cc:1428] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 08:19:49.250937 14658 tablet_bootstrap.cc:654] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:49.252331 14658 tablet_bootstrap.cc:492] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: No bootstrap required, opened a new log
I20260808 08:19:49.252429 14658 ts_tablet_manager.cc:1397] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:19:49.252666 14657 tablet_bootstrap.cc:492] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: No bootstrap required, opened a new log
I20260808 08:19:49.252758 14657 ts_tablet_manager.cc:1397] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260808 08:19:49.253185 14657 raft_consensus.cc:348] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.253271 14657 raft_consensus.cc:374] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:49.253293 14657 raft_consensus.cc:729] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1671e072b434dbcb441e05ea30c85d5, State: Initialized, Role: FOLLOWER
I20260808 08:19:49.253228 14658 raft_consensus.cc:348] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.253357 14658 raft_consensus.cc:374] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:49.253393 14658 raft_consensus.cc:729] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e34c1f115c34022af6b10a30978601a, State: Initialized, Role: FOLLOWER
I20260808 08:19:49.253450 14657 consensus_queue.cc:260] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.253625 14658 consensus_queue.cc:260] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.253713 14657 ts_tablet_manager.cc:1428] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:49.253934 14664 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.253995 14658 ts_tablet_manager.cc:1428] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:19:49.254073 14664 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.254293 14667 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.254420 14667 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.321136 14667 raft_consensus.cc:482] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:49.321300 14667 raft_consensus.cc:504] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.321821 14667 leader_election.cc:288] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145), d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661)
I20260808 08:19:49.322355 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b506f4153d43b6b37a321c5e3b3509" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c58cd846f5477a8ab695282ccb62e2" is_pre_election: true
I20260808 08:19:49.322376 14353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b506f4153d43b6b37a321c5e3b3509" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1671e072b434dbcb441e05ea30c85d5" is_pre_election: true
I20260808 08:19:49.322527 14353 raft_consensus.cc:2454] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 0.
I20260808 08:19:49.322527 14165 raft_consensus.cc:2454] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 0.
I20260808 08:19:49.322820 14459 leader_election.cc:302] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e34c1f115c34022af6b10a30978601a, d1671e072b434dbcb441e05ea30c85d5; no voters:
I20260808 08:19:49.323016 14667 raft_consensus.cc:2781] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:49.323078 14667 raft_consensus.cc:482] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:49.323125 14667 raft_consensus.cc:3037] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:49.324272 14667 raft_consensus.cc:504] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.324620 14667 leader_election.cc:288] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 election: Requested vote from peers 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145), d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661)
I20260808 08:19:49.325058 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b506f4153d43b6b37a321c5e3b3509" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c58cd846f5477a8ab695282ccb62e2"
I20260808 08:19:49.325088 14353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b506f4153d43b6b37a321c5e3b3509" candidate_uuid: "6e34c1f115c34022af6b10a30978601a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1671e072b434dbcb441e05ea30c85d5"
I20260808 08:19:49.325194 14165 raft_consensus.cc:3037] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:49.325206 14353 raft_consensus.cc:3037] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:49.326485 14353 raft_consensus.cc:2454] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 1.
I20260808 08:19:49.326514 14165 raft_consensus.cc:2454] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e34c1f115c34022af6b10a30978601a in term 1.
I20260808 08:19:49.326840 14459 leader_election.cc:302] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c58cd846f5477a8ab695282ccb62e2, 6e34c1f115c34022af6b10a30978601a; no voters:
I20260808 08:19:49.327014 14667 raft_consensus.cc:2781] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:49.327111 14667 raft_consensus.cc:686] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [term 1 LEADER]: Becoming Leader. State: Replica: 6e34c1f115c34022af6b10a30978601a, State: Running, Role: LEADER
I20260808 08:19:49.327273 14667 consensus_queue.cc:237] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } }
I20260808 08:19:49.327946 14665 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: TxnStatusTablet state changed. Reason: New leader 6e34c1f115c34022af6b10a30978601a. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.328063 14665 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:19:49.328451 14728 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 08:19:49.328704 14728 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 08:19:49.329196 13939 catalog_manager.cc:5495] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e34c1f115c34022af6b10a30978601a (127.12.13.3). New cstate: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } health_report { overall_health: HEALTHY } } }
I20260808 08:19:49.361063 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.357130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:43133 (local address 127.0.0.1:48770)
0808 08:19:49.357280 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:49.357299 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:49.357499 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:49.357810 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:49.357813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:49.357824 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:49.357974 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.357987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.358937 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.358942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.359752 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.359763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.360244 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.360248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.360358 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.360413 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.360752 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.360772 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 08:19:49.360840 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 08:19:49.360935 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 08:19:49.361063 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.357257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.12.13.3:43133)
0808 08:19:49.357399 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.357404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.357420 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:19:49.357637 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.357640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.357698 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.357784 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:49.358132 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.358813 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.359884 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.360125 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.360241 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.360573 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.360661 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.360745 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.360856 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.360912 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:19:49.365006 14165 raft_consensus.cc:1261] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Refusing update from remote peer 6e34c1f115c34022af6b10a30978601a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:49.365170 14353 raft_consensus.cc:1261] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Refusing update from remote peer 6e34c1f115c34022af6b10a30978601a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:49.365561 14667 consensus_queue.cc:1035] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:49.365722 14665 consensus_queue.cc:1035] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:49.367251 14664 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: TxnStatusTablet state changed. Reason: New leader 6e34c1f115c34022af6b10a30978601a. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.367368 14664 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.367298 14663 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: TxnStatusTablet state changed. Reason: New leader 6e34c1f115c34022af6b10a30978601a. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.367420 14663 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.370003 14667 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.370128 14667 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:19:49.370478 14675 mvcc.cc:205] Tried to move back new op lower bound from 7316181767633580032 to 7316181767485665280. Current Snapshot: MvccSnapshot[applied={T|T < 7316181767633580032}]
I20260808 08:19:49.370654 14663 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.370745 14663 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 15c58cd846f5477a8ab695282ccb62e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.372951 14665 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.373056 14665 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P 6e34c1f115c34022af6b10a30978601a: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:19:49.374014 14664 tablet_replica.cc:439] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e34c1f115c34022af6b10a30978601a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 } } peers { permanent_uuid: "d1671e072b434dbcb441e05ea30c85d5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40661 } } peers { permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 } } }
I20260808 08:19:49.374128 14664 tablet_replica.cc:442] T f1b506f4153d43b6b37a321c5e3b3509 P d1671e072b434dbcb441e05ea30c85d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:49.388126 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.383283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.12.13.2:40661)
0808 08:19:49.383693 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.383698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.383711 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:19:49.383800 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.383804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.383877 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.383987 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:49.384321 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.385114 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.386398 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.386744 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.386862 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.387133 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.387679 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.387818 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.387919 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.387983 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260808 08:19:49.400969 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.397331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:50975 (local address 127.12.13.3:43133)
0808 08:19:49.397464 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.397469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.397488 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:19:49.397536 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.397538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.397582 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.397646 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:49.397951 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.398627 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.399908 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.400116 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.400199 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.400528 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.400599 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.400654 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.400789 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.400859 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:19:49.401028 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.396881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:43133 (local address 127.12.13.2:50975)
0808 08:19:49.397235 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:49.397251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:49.397354 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:49.397689 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:49.397692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:49.397701 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:49.397825 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.397831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.398734 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.398738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.399769 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.399780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.400236 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.400240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.400330 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.400390 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.400689 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.400695 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:19:49.400765 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 08:19:49.400855 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
I20260808 08:19:49.410321 14165 raft_consensus.cc:1261] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Refusing update from remote peer d1671e072b434dbcb441e05ea30c85d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:49.410431 14563 raft_consensus.cc:1261] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Refusing update from remote peer d1671e072b434dbcb441e05ea30c85d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:49.410889 14664 consensus_queue.cc:1035] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c58cd846f5477a8ab695282ccb62e2" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:49.411092 14699 consensus_queue.cc:1035] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e34c1f115c34022af6b10a30978601a" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:49.436153 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.432176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40661 (local address 127.12.13.3:57147)
0808 08:19:49.432535 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:49.432553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:49.432658 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:49.432904 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:49.432906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:49.432911 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:49.433043 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.433049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.434023 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.434028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.434880 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:49.434890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.435446 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.435449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:49.435507 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.435550 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.435913 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.435918 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 08:19:49.435968 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 08:19:49.436033 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":126,"threads_started":1}
I20260808 08:19:49.436179 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.432245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:57147 (local address 127.12.13.2:40661)
0808 08:19:49.432364 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.432368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.432610 (+ 242us) server_negotiation.cc:407] Connection header received
0808 08:19:49.432754 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.432757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.432816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.432903 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:49.433159 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.433926 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.435000 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.435324 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.435437 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.435685 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:49.435769 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:49.435904 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.435983 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.436035 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 08:19:49.465021 13934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:49.465173 13934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:49.467571 13934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4c99474d9ae44daa6524afcd145fabb: Sending DeleteTablet for 3 replicas of tablet 83d99c6af35b41faade8ff55b8e7b349
I20260808 08:19:49.468297 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d99c6af35b41faade8ff55b8e7b349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:48748
I20260808 08:19:49.468513 14682 tablet_replica.cc:330] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: stopping tablet replica
I20260808 08:19:49.468683 14682 raft_consensus.cc:2229] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:49.468829 14682 raft_consensus.cc:2258] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:49.469043 14138 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d99c6af35b41faade8ff55b8e7b349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:47850
I20260808 08:19:49.469239 14686 tablet_replica.cc:330] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: stopping tablet replica
I20260808 08:19:49.469377 14686 raft_consensus.cc:2229] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:49.469530 14686 raft_consensus.cc:2258] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:49.470484 14686 ts_tablet_manager.cc:1905] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:49.470484 14682 ts_tablet_manager.cc:1905] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:49.473539 14682 ts_tablet_manager.cc:1918] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:49.473539 14686 ts_tablet_manager.cc:1918] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:49.473614 14682 log.cc:1196] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/83d99c6af35b41faade8ff55b8e7b349
I20260808 08:19:49.473614 14686 log.cc:1196] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/83d99c6af35b41faade8ff55b8e7b349
I20260808 08:19:49.473984 14686 ts_tablet_manager.cc:1939] T 83d99c6af35b41faade8ff55b8e7b349 P 15c58cd846f5477a8ab695282ccb62e2: Deleting consensus metadata
I20260808 08:19:49.474524 14332 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d99c6af35b41faade8ff55b8e7b349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:49540
I20260808 08:19:49.474746 14681 tablet_replica.cc:330] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: stopping tablet replica
I20260808 08:19:49.474898 14681 raft_consensus.cc:2229] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:19:49.475214 14681 raft_consensus.cc:2258] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:49.475840 13918 catalog_manager.cc:4841] TS 15c58cd846f5477a8ab695282ccb62e2 (127.12.13.1:39145): tablet 83d99c6af35b41faade8ff55b8e7b349 (table test-table [id=fc1468269aed4f01a7e7363ffbef997d]) successfully deleted
I20260808 08:19:49.475980 14681 ts_tablet_manager.cc:1905] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:49.477231 14682 ts_tablet_manager.cc:1939] T 83d99c6af35b41faade8ff55b8e7b349 P 6e34c1f115c34022af6b10a30978601a: Deleting consensus metadata
I20260808 08:19:49.479068 14681 ts_tablet_manager.cc:1918] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:49.479133 14681 log.cc:1196] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/83d99c6af35b41faade8ff55b8e7b349
I20260808 08:19:49.479336 14681 ts_tablet_manager.cc:1939] T 83d99c6af35b41faade8ff55b8e7b349 P d1671e072b434dbcb441e05ea30c85d5: Deleting consensus metadata
I20260808 08:19:49.479781 13919 catalog_manager.cc:4841] TS 6e34c1f115c34022af6b10a30978601a (127.12.13.3:43133): tablet 83d99c6af35b41faade8ff55b8e7b349 (table test-table [id=fc1468269aed4f01a7e7363ffbef997d]) successfully deleted
I20260808 08:19:49.480330 13918 catalog_manager.cc:4841] TS d1671e072b434dbcb441e05ea30c85d5 (127.12.13.2:40661): tablet 83d99c6af35b41faade8ff55b8e7b349 (table test-table [id=fc1468269aed4f01a7e7363ffbef997d]) successfully deleted
Aug 08 08:19:49 dist-test-slave-6ntk krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:19:49.491518 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.478396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.12.13.1:39145)
0808 08:19:49.478533 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.478538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.480276 (+ 1738us) server_negotiation.cc:407] Connection header received
0808 08:19:49.480556 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.480559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.480622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.480678 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:49.481513 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.482146 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.483332 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.483670 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.483795 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.487341 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:49.487362 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:49.487379 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:49.487410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:49.489266 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:49.489785 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:49.489792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:49.489793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:49.489937 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:49.490385 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:49.490389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:49.490391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:49.490676 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:49.490928 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.491147 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.491281 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 08:19:49.492100 14218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862
I20260808 08:19:49.502568 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:49.494129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.12.13.62:43405)
0808 08:19:49.494311 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:49.494316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:49.494756 (+ 440us) server_negotiation.cc:407] Connection header received
0808 08:19:49.494875 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:49.494878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:49.494926 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:49.495023 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:49.495686 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.496226 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.497028 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:49.497289 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:49.497405 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:49.499077 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:49.499081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:49.499085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:49.499116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:49.500708 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:49.501163 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:49.501167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:49.501169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:49.501263 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:49.501551 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:49.501555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:49.501556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:49.501767 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:49.501955 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:49.502213 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 08:19:49.502359 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 08:19:49.502997 13934 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41866
I20260808 08:19:49.504758 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 14046
I20260808 08:19:49.520499 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 14236
I20260808 08:19:49.530259 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 14440
I20260808 08:19:49.545078 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13893
2026-08-08T08:19:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4159 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:19:49 dist-test-slave-6ntk krb5kdc[14764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:19:49 dist-test-slave-6ntk krb5kdc[14764](info): set up 2 sockets
Aug 08 08:19:49 dist-test-slave-6ntk krb5kdc[14764](info): commencing operation
krb5kdc: starting...
W20260808 08:19:50.861624 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:19:50 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:19:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:19:51.007064 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35727
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:51.128906 14780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:51.129259 14780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:51.129302 14780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:51.133359 14780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:19:51.133433 14780 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 08:19:51.133451 14780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:51.133469 14780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:51.133486 14780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:19:51.133502 14780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:19:51.139218 14780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:35727
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:51.140439 14780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:51.141518 14780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:19:51.151101 14780 server_base.cc:1034] running on GCE node
W20260808 08:19:51.151046 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:51.151048 14792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:51.151059 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:51.152153 14780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:51.153487 14780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:51.154733 14780 hybrid_clock.cc:648] HybridClock initialized: now 1786177191154687 us; error 63 us; skew 500 ppm
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:51.161430 14780 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:19:51.162545 14780 webserver.cc:460] Webserver started at http://127.12.13.62:43787/ using document root <none> and password file <none>
I20260808 08:19:51.163200 14780 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:51.163305 14780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:51.163573 14780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:51.165859 14780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "141d2298af304f79b68e641cc96d1460"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "964d0a27648affd07008058bbe8f5db0"
server_key_iv: "200a7be4b6aed7b7cf102bb8164b122e"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.166621 14780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "141d2298af304f79b68e641cc96d1460"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "964d0a27648affd07008058bbe8f5db0"
server_key_iv: "200a7be4b6aed7b7cf102bb8164b122e"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.170827 14780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:19:51.173651 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.175271 14780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:51.175438 14780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "141d2298af304f79b68e641cc96d1460"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "964d0a27648affd07008058bbe8f5db0"
server_key_iv: "200a7be4b6aed7b7cf102bb8164b122e"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.175578 14780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:51.189343 14780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:51.193192 14780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:51.193451 14780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:51.203106 14780 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:35727
I20260808 08:19:51.203140 14894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:35727 every 8 connection(s)
I20260808 08:19:51.204257 14780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:19:51.206103 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 14780
I20260808 08:19:51.206238 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:19:51.206535 12340 external_mini_cluster.cc:1428] Setting key bc67200d4ea0d5fa5a222fa194a5779a
I20260808 08:19:51.208045 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:51.214396 14896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Bootstrap starting.
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:51.217028 14896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:51.218138 14896 log.cc:824] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:51.220883 14896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: No bootstrap required, opened a new log
I20260808 08:19:51.223464 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.208223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.12.13.62:35727)
0808 08:19:51.208731 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:51.208748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:51.208789 (+ 41us) server_negotiation.cc:407] Connection header received
0808 08:19:51.209493 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:51.209513 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:51.210003 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:51.210394 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:51.211342 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.212368 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.213291 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.213804 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.214053 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.216709 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:51.216716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:51.216734 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:51.216781 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:51.219912 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.220415 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.220422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.220428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.220593 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.220897 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.220903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.220907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.221477 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:51.221667 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:51.222107 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 08:19:51.222430 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":153,"threads_started":1}
I20260808 08:19:51.225059 14896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141d2298af304f79b68e641cc96d1460" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35727 } }
I20260808 08:19:51.225298 14896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:51.225337 14896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141d2298af304f79b68e641cc96d1460, State: Initialized, Role: FOLLOWER
I20260808 08:19:51.225955 14896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141d2298af304f79b68e641cc96d1460" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35727 } }
I20260808 08:19:51.226127 14896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:19:51.226204 14896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:19:51.226318 14896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:51.227321 14896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141d2298af304f79b68e641cc96d1460" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35727 } }
I20260808 08:19:51.227695 14896 leader_election.cc:302] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 141d2298af304f79b68e641cc96d1460; no voters:
I20260808 08:19:51.228040 14896 leader_election.cc:288] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:19:51.228194 14902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:51.228471 14902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [term 1 LEADER]: Becoming Leader. State: Replica: 141d2298af304f79b68e641cc96d1460, State: Running, Role: LEADER
I20260808 08:19:51.228803 14902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141d2298af304f79b68e641cc96d1460" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35727 } }
I20260808 08:19:51.229411 14896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:19:51.230880 14906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 141d2298af304f79b68e641cc96d1460. Latest consensus state: current_term: 1 leader_uuid: "141d2298af304f79b68e641cc96d1460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141d2298af304f79b68e641cc96d1460" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35727 } } }
I20260808 08:19:51.231053 14906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:51.230890 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "141d2298af304f79b68e641cc96d1460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141d2298af304f79b68e641cc96d1460" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35727 } } }
I20260808 08:19:51.231161 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460 [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:51.231544 14925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:19:51.235038 14925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:19:51.241961 14925 catalog_manager.cc:1347] Generated new cluster ID: 78db706e0d5a41be9ff9b3611815dfb5
I20260808 08:19:51.242046 14925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:19:51.254688 14925 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:19:51.255743 14925 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:19:51.271620 14925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Generated new TSK 0
I20260808 08:19:51.272440 14925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:19:51.314631 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35727
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:51.438680 14939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:51.438983 14939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:51.439033 14939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:51.443281 14939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:51.443348 14939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:51.443390 14939 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 08:19:51.443449 14939 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:19:51.443507 14939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:19:51.449048 14939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:35727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:51.450358 14939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:51.451525 14939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:51.462051 14949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:51.462107 14939 server_base.cc:1034] running on GCE node
W20260808 08:19:51.462173 14953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:51.462054 14948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:51.462826 14939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:51.463467 14939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:51.464639 14939 hybrid_clock.cc:648] HybridClock initialized: now 1786177191464614 us; error 40 us; skew 500 ppm
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:51.467551 14939 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:19:51.468454 14939 webserver.cc:460] Webserver started at http://127.12.13.1:34927/ using document root <none> and password file <none>
I20260808 08:19:51.469077 14939 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:51.469179 14939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:51.469439 14939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:51.471621 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "fc8a11ed501048f6a61dd5c71e2d1657"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "7a768021f940000a0bd71ac09d6c68f9"
server_key_iv: "c428d4a92ce8d3a7bfdaf56aa800ef15"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.472218 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "fc8a11ed501048f6a61dd5c71e2d1657"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "7a768021f940000a0bd71ac09d6c68f9"
server_key_iv: "c428d4a92ce8d3a7bfdaf56aa800ef15"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.476310 14939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 08:19:51.478982 14960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.480279 14939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:51.480440 14939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "fc8a11ed501048f6a61dd5c71e2d1657"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "7a768021f940000a0bd71ac09d6c68f9"
server_key_iv: "c428d4a92ce8d3a7bfdaf56aa800ef15"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.480566 14939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:51.490442 14939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:51.493213 14939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:51.493435 14939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:51.495010 14939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:51.495092 14939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.495164 14939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:51.495199 14939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:51.507172 14939 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:34703
I20260808 08:19:51.507252 15142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:34703 every 8 connection(s)
I20260808 08:19:51.508353 14939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:19:51.510998 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.495882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:46997 (local address 127.12.13.62:35727)
0808 08:19:51.496020 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:51.496023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:51.497135 (+ 1112us) server_negotiation.cc:407] Connection header received
0808 08:19:51.498310 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:51.498315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:51.498387 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:51.498452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:51.500347 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.501285 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.503019 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.503281 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.503419 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.506725 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:51.506729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:51.506732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:51.506767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:51.508517 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.509232 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.509236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.509238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.509356 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.509790 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.509796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.509800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.510085 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:51.510171 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:51.510713 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 08:19:51.510865 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:19:51.511875 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.496447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.12.13.1:46997)
0808 08:19:51.496971 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:51.497016 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:51.498028 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:51.498946 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:51.498958 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:51.499525 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:51.500173 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.500193 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.501434 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.501439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.502904 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.502915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.503475 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.503479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.503664 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.504325 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:51.504343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:51.506461 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:51.508657 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.508663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.508680 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.509093 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.509486 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.509489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.509491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.509645 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.510181 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:51.510187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:51.510449 (+ 262us) client_negotiation.cc:769] Sending connection context
0808 08:19:51.510611 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 08:19:51.510899 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":148,"threads_started":1}
I20260808 08:19:51.513041 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 14939
I20260808 08:19:51.513190 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:19:51.513499 12340 external_mini_cluster.cc:1428] Setting key 505caa0bd36a2a2021fd30eab74642d3
I20260808 08:19:51.513646 15145 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35727
I20260808 08:19:51.513813 15145 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:51.514523 15145 heartbeater.cc:511] Master 127.12.13.62:35727 requested a full tablet report, sending...
I20260808 08:19:51.516207 14831 ts_manager.cc:194] Registered new tserver with Master: fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703)
I20260808 08:19:51.517659 14831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:46997
I20260808 08:19:51.526237 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.516241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.12.13.1:51229)
0808 08:19:51.518068 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:51.518081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:51.518216 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:51.518485 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:51.518488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:51.518748 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:51.518896 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.518906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.520038 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.520041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.520669 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.520679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.521348 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.521352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.521432 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.522112 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:51.522124 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:51.522494 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:51.524556 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.524558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.524560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.524763 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.525083 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.525085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.525086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.525166 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.525672 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:51.525674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:51.525759 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 08:19:51.525863 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 08:19:51.526071 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:19:51.526309 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.516527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:51229 (local address 127.12.13.62:35727)
0808 08:19:51.516711 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:51.516717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:51.518169 (+ 1452us) server_negotiation.cc:407] Connection header received
0808 08:19:51.518316 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:51.518319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:51.518376 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:51.518464 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:51.519051 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.519874 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.520816 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.521211 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.521355 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.522663 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:51.522668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:51.522674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:51.522712 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:51.524430 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.524874 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.524878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.524880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.524979 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.525268 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.525273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.525274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.525554 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:51.525685 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:51.525991 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 08:19:51.526122 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:19:51.550498 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35727
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:51.678690 15157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:51.678960 15157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:51.679008 15157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:51.683033 15157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:51.683096 15157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:51.683138 15157 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 08:19:51.683153 15157 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:19:51.683185 15157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:19:51.688637 15157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:35727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:51.689682 15157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:51.690779 15157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:51.700955 15169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:51.700973 15168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:51.700973 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:51.701231 15157 server_base.cc:1034] running on GCE node
I20260808 08:19:51.701757 15157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:51.702435 15157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:51.703644 15157 hybrid_clock.cc:648] HybridClock initialized: now 1786177191703626 us; error 37 us; skew 500 ppm
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:51.706687 15157 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:19:51.707638 15157 webserver.cc:460] Webserver started at http://127.12.13.2:38711/ using document root <none> and password file <none>
I20260808 08:19:51.708316 15157 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:51.708395 15157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:51.708633 15157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:51.710916 15157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "d13b42f89788467ba9404259804c74d3"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "1e6c25ddcd073ad483e7bd514eb1fff3"
server_key_iv: "10f63a4b75d96e9dc65a5505ed893ced"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.711524 15157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "d13b42f89788467ba9404259804c74d3"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "1e6c25ddcd073ad483e7bd514eb1fff3"
server_key_iv: "10f63a4b75d96e9dc65a5505ed893ced"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.715893 15157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:19:51.718658 15183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.720010 15157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:51.720167 15157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "d13b42f89788467ba9404259804c74d3"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "1e6c25ddcd073ad483e7bd514eb1fff3"
server_key_iv: "10f63a4b75d96e9dc65a5505ed893ced"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.720301 15157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:51.746563 15157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:51.750061 15157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:51.750286 15157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:51.751910 15157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:51.752003 15157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.752070 15157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:51.752125 15157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:51.763447 15157 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:45127
I20260808 08:19:51.763518 15353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:45127 every 8 connection(s)
I20260808 08:19:51.764714 15157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:19:51.767732 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.753082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:49843 (local address 127.12.13.62:35727)
0808 08:19:51.753271 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:51.753276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:51.754217 (+ 941us) server_negotiation.cc:407] Connection header received
0808 08:19:51.755443 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:51.755447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:51.755529 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:51.755668 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:51.757540 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.758559 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.760056 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.760301 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.760422 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.763673 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:51.763677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:51.763680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:51.763710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:51.765486 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.766190 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.766193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.766194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.766286 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.766584 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.766587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.766588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.766789 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:51.766869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:51.767406 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 08:19:51.767571 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 08:19:51.768622 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.753466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.12.13.2:49843)
0808 08:19:51.754029 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:51.754087 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:51.755095 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:51.755930 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:51.755942 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:51.756503 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:51.757341 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.757367 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.758731 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.758736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.759891 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.759902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.760485 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.760489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.760681 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.761406 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:51.761437 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:51.763485 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:51.765633 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.765639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.765654 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.766051 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.766361 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.766363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.766364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.766493 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.766904 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:51.766910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:51.767153 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 08:19:51.767308 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 08:19:51.767734 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":158,"threads_started":1}
I20260808 08:19:51.769460 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 15157
I20260808 08:19:51.769578 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:19:51.769923 12340 external_mini_cluster.cc:1428] Setting key 34460ff7e72d10fea9cd977b649bd5d9
I20260808 08:19:51.770273 15354 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35727
I20260808 08:19:51.770496 15354 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:51.771162 15354 heartbeater.cc:511] Master 127.12.13.62:35727 requested a full tablet report, sending...
I20260808 08:19:51.774235 14834 ts_manager.cc:194] Registered new tserver with Master: d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:51.775136 14834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:49843
I20260808 08:19:51.780596 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.772564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.12.13.2:43395)
0808 08:19:51.772966 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:51.772978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:51.773101 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:51.773374 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:51.773377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:51.773593 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:51.773691 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.773697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.774579 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.774583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.775161 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:51.775169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.775769 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.775773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:51.775882 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.776493 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:51.776506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:51.776925 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:51.778821 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.778826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.778836 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.779043 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.779412 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:51.779414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:51.779415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:51.779545 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:51.780003 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:51.780007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:51.780107 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 08:19:51.780221 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 08:19:51.780388 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260808 08:19:51.780604 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.772656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:43395 (local address 127.12.13.62:35727)
0808 08:19:51.772801 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:51.772806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:51.773057 (+ 251us) server_negotiation.cc:407] Connection header received
0808 08:19:51.773202 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:51.773205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:51.773259 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:51.773359 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:51.773815 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.774440 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.775353 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:51.775639 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:51.775753 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:51.777065 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:51.777069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:51.777072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:51.777097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:51.778693 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.779156 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.779160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.779162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.779249 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:51.779646 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:51.779649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:51.779651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:51.779883 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:51.780001 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:51.780281 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 08:19:51.780423 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:19:51.804387 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35727
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:51.923892 15369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:51.924214 15369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:51.924263 15369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:51.928434 15369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:51.928507 15369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:51.928550 15369 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 08:19:51.928565 15369 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:19:51.928596 15369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:19:51.934258 15369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:35727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:51.935395 15369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:51.936442 15369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:51.946600 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:51.946594 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:51.946816 15369 server_base.cc:1034] running on GCE node
W20260808 08:19:51.946870 15380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:51.947319 15369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:51.948031 15369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:51.949239 15369 hybrid_clock.cc:648] HybridClock initialized: now 1786177191949224 us; error 32 us; skew 500 ppm
Aug 08 08:19:51 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:51.952244 15369 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:19:51.953190 15369 webserver.cc:460] Webserver started at http://127.12.13.3:42639/ using document root <none> and password file <none>
I20260808 08:19:51.953804 15369 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:51.953949 15369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:51.954214 15369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:51.956373 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "32fbb180f4204485ae19185fb8e59c96"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "f7e39bb03077975b0a70be7b6c838d62"
server_key_iv: "0072c45c9c67a43ad3bff1e5eeb246d6"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.956979 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "32fbb180f4204485ae19185fb8e59c96"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "f7e39bb03077975b0a70be7b6c838d62"
server_key_iv: "0072c45c9c67a43ad3bff1e5eeb246d6"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.961205 15369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 08:19:51.963889 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.965171 15369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 08:19:51.965334 15369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "32fbb180f4204485ae19185fb8e59c96"
format_stamp: "Formatted at 2026-08-08 08:19:51 on dist-test-slave-6ntk"
server_key: "f7e39bb03077975b0a70be7b6c838d62"
server_key_iv: "0072c45c9c67a43ad3bff1e5eeb246d6"
server_key_version: "encryptionkey@0"
I20260808 08:19:51.965472 15369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:51.992681 15369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:51.995817 15369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:51.996057 15369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:51.997641 15369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:51.997728 15369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:51.997803 15369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:51.997860 15369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:19:52 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:52.009204 15369 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:44227
I20260808 08:19:52.009268 15556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:44227 every 8 connection(s)
I20260808 08:19:52.010294 15369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:19:52.012132 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 15369
I20260808 08:19:52.012228 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:19:52.012621 12340 external_mini_cluster.cc:1428] Setting key ddc9b19a1a5dbd71205a945146a9a748
I20260808 08:19:52.013689 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.999051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.12.13.3:53739)
0808 08:19:51.999545 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:51.999599 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.000650 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.001385 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.001392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.001814 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:52.002542 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.002559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.003772 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.003776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.004934 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.004945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.005501 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.005505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.005687 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.006460 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:52.006485 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:52.008272 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:52.010499 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:52.010504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:52.010517 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:52.010828 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:52.011168 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:52.011171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:52.011173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:52.011373 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:52.011853 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:52.011858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:52.012090 (+ 232us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.012296 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.012688 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260808 08:19:52.014454 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:51.998600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:53739 (local address 127.12.13.62:35727)
0808 08:19:51.998792 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:51.998800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:51.999723 (+ 923us) server_negotiation.cc:407] Connection header received
0808 08:19:52.000963 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.000969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.001045 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.001156 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:52.002696 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.003608 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.005073 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.005315 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.005444 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.008469 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:52.008473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:52.008476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:52.008506 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:52.010327 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.010957 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.010960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.010962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.011059 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.011518 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.011521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.011522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.011734 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:52.011826 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.014151 (+ 2325us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.014291 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 08:19:52.015443 15559 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35727
I20260808 08:19:52.015591 15559 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:52.016139 15559 heartbeater.cc:511] Master 127.12.13.62:35727 requested a full tablet report, sending...
I20260808 08:19:52.017179 14834 ts_manager.cc:194] Registered new tserver with Master: 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227)
I20260808 08:19:52.017920 14834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:53739
I20260808 08:19:52.024492 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.017708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:34589 (local address 127.12.13.62:35727)
0808 08:19:52.017828 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.017832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.018055 (+ 223us) server_negotiation.cc:407] Connection header received
0808 08:19:52.018183 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.018189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.018245 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.018327 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:52.018813 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.019314 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.020137 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.020365 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.020479 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.021503 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:52.021507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:52.021510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:52.021540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:52.022812 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.023254 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.023259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.023261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.023365 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.023674 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.023677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.023679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.023860 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:52.023969 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.024239 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.024351 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:19:52.024497 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.017596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.12.13.3:34589)
0808 08:19:52.017959 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.017976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.018074 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.018355 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.018358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.018583 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:52.018690 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.018696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.019443 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.019446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.020010 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.020018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.020482 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.020485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.020576 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.021059 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:52.021070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:52.021367 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:52.022948 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:52.022951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:52.022953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:52.023151 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:52.023462 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:52.023466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:52.023468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:52.023559 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:52.023970 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:52.023974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:52.024069 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.024181 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.024319 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
I20260808 08:19:52.026768 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 08:19:52 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 08:19:52 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:52.055783 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.046821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.12.13.62:35727)
0808 08:19:52.046988 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.046992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.047005 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:19:52.047052 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.047055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.047097 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.047170 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:52.047996 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.048535 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.049402 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.049622 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.049705 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.052365 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:52.052369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:52.052371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:52.052399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:52.054069 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.054518 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.054521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.054522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.054612 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.054878 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.054881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.054882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.055070 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:52.055178 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.055425 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.055536 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 08:19:52.059367 14834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:19:52.062822 14834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:52.079160 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.072952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34703 (local address 127.0.0.1:40158)
0808 08:19:52.074034 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.074099 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.074245 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.075072 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.075079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.075100 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.075244 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.075251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.076489 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.076493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.077574 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.077584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.078498 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.078501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.078655 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.078720 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.078869 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.078997 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":13,"thread_start_us":87,"threads_started":1}
I20260808 08:19:52.079202 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.073094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.12.13.1:34703)
0808 08:19:52.074545 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.074551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.074573 (+ 22us) server_negotiation.cc:407] Connection header received
0808 08:19:52.074642 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.074647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.074832 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.075021 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.075383 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.076361 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.077719 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.078360 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.078486 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.078656 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.078969 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.079075 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":100,"threads_started":1}
I20260808 08:19:52.080668 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.073859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.12.13.2:45127)
0808 08:19:52.074684 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.074695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.074952 (+ 257us) server_negotiation.cc:407] Connection header received
0808 08:19:52.075041 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.075046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.075232 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.075392 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.075667 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.076915 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.078810 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.079695 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.079798 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.079949 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.080397 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.080484 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":121,"threads_started":1}
I20260808 08:19:52.081094 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.073392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45127 (local address 127.0.0.1:51444)
0808 08:19:52.074821 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.074835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.074934 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.075419 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.075421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.075445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.075534 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.075539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.077071 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.077075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.078648 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.078662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.079954 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.079958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.080052 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.080083 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.080846 (+ 763us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.080943 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"mutex_wait_us":138,"thread_start_us":112,"threads_started":1}
I20260808 08:19:52.081535 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.073081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44227 (local address 127.0.0.1:59958)
0808 08:19:52.074034 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.074099 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.074245 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.075228 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.075231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.075254 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.075391 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.075399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.077071 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.077075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.078681 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.078690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.081122 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.081126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.081223 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.081247 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.081319 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.081405 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":158,"spinlock_wait_cycles":768,"thread_start_us":122,"threads_started":1}
I20260808 08:19:52.082007 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.074260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.12.13.3:44227)
0808 08:19:52.074581 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.074594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.074620 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:19:52.074706 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.074713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.074944 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.075098 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.075541 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.076914 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.080014 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.080784 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.080895 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.081054 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.081731 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.081832 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":126,"threads_started":1}
I20260808 08:19:52.082806 15042 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0e82798fb745499c5f9de4b9a942c8 (DEFAULT_TABLE table=test-table [id=b2c7be6420ef4143b0afd1174572f39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:52.084084 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0e82798fb745499c5f9de4b9a942c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.084779 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0e82798fb745499c5f9de4b9a942c8 (DEFAULT_TABLE table=test-table [id=b2c7be6420ef4143b0afd1174572f39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:52.084779 15254 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0e82798fb745499c5f9de4b9a942c8 (DEFAULT_TABLE table=test-table [id=b2c7be6420ef4143b0afd1174572f39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:52.085942 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0e82798fb745499c5f9de4b9a942c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.085958 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0e82798fb745499c5f9de4b9a942c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.090996 15593 tablet_bootstrap.cc:492] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Bootstrap starting.
I20260808 08:19:52.093209 15593 tablet_bootstrap.cc:654] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.094296 15596 tablet_bootstrap.cc:492] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Bootstrap starting.
I20260808 08:19:52.094775 15594 tablet_bootstrap.cc:492] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Bootstrap starting.
I20260808 08:19:52.095108 15593 log.cc:824] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:52.097064 15593 tablet_bootstrap.cc:492] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: No bootstrap required, opened a new log
I20260808 08:19:52.097174 15594 tablet_bootstrap.cc:654] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.097407 15593 ts_tablet_manager.cc:1397] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 08:19:52.097600 15596 tablet_bootstrap.cc:654] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.098348 15594 log.cc:824] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:52.098404 15596 log.cc:824] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Log is configured to *not* fsync() on all Append() calls
I20260808 08:19:52.100258 15596 tablet_bootstrap.cc:492] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: No bootstrap required, opened a new log
I20260808 08:19:52.100503 15596 ts_tablet_manager.cc:1397] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:19:52.100613 15594 tablet_bootstrap.cc:492] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: No bootstrap required, opened a new log
I20260808 08:19:52.100818 15594 ts_tablet_manager.cc:1397] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:19:52.102414 15593 raft_consensus.cc:348] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.102753 15593 raft_consensus.cc:374] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.102841 15593 raft_consensus.cc:729] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.103350 15593 consensus_queue.cc:260] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.104151 15145 heartbeater.cc:503] Master 127.12.13.62:35727 was elected leader, sending a full tablet report...
I20260808 08:19:52.104519 15593 ts_tablet_manager.cc:1428] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 08:19:52.104344 15596 raft_consensus.cc:348] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.104595 15596 raft_consensus.cc:374] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.104645 15596 raft_consensus.cc:729] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.104899 15594 raft_consensus.cc:348] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.105132 15594 raft_consensus.cc:374] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.105183 15594 raft_consensus.cc:729] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b42f89788467ba9404259804c74d3, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.105116 15596 consensus_queue.cc:260] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.105674 15594 consensus_queue.cc:260] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.105813 15559 heartbeater.cc:503] Master 127.12.13.62:35727 was elected leader, sending a full tablet report...
I20260808 08:19:52.106231 15596 ts_tablet_manager.cc:1428] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260808 08:19:52.106840 15354 heartbeater.cc:503] Master 127.12.13.62:35727 was elected leader, sending a full tablet report...
I20260808 08:19:52.106945 15594 ts_tablet_manager.cc:1428] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:19:52.132665 15600 raft_consensus.cc:482] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:52.132900 15600 raft_consensus.cc:504] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.134428 15600 leader_election.cc:288] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227), d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:52.138492 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.134815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56149 (local address 127.12.13.3:44227)
0808 08:19:52.134972 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.134977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.135169 (+ 192us) server_negotiation.cc:407] Connection header received
0808 08:19:52.135341 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.135344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.135402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.135490 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.135782 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.136316 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.137407 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.137824 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.138001 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.138071 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.138280 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.138356 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 08:19:52.138518 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.134833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40469 (local address 127.12.13.2:45127)
0808 08:19:52.134972 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.134977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.135278 (+ 301us) server_negotiation.cc:407] Connection header received
0808 08:19:52.135442 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.135445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.135497 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.135581 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.135876 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.136405 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.137428 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.137828 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.137994 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.138073 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.138313 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.138394 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:19:52.138527 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.134651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44227 (local address 127.12.13.1:56149)
0808 08:19:52.135058 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.135071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.135213 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.135505 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.135508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.135525 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.135658 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.135665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.136446 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.136449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.137259 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.137269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.138005 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.138009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.138138 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.138166 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.138263 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.138346 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":4,"thread_start_us":158,"threads_started":1}
I20260808 08:19:52.138535 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.134651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45127 (local address 127.12.13.1:40469)
0808 08:19:52.135156 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.135168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.135317 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.135605 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.135608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.135624 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.135753 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.135759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.136526 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.136533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.137289 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.137298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.138005 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.138008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.138180 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.138199 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.138296 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.138366 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":78,"thread_start_us":143,"threads_started":1}
I20260808 08:19:52.139209 15499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0e82798fb745499c5f9de4b9a942c8" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fbb180f4204485ae19185fb8e59c96" is_pre_election: true
I20260808 08:19:52.139348 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0e82798fb745499c5f9de4b9a942c8" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3" is_pre_election: true
I20260808 08:19:52.139566 15499 raft_consensus.cc:2454] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 0.
I20260808 08:19:52.139667 15285 raft_consensus.cc:2454] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 0.
I20260808 08:19:52.140101 14967 leader_election.cc:302] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:52.140432 15600 raft_consensus.cc:2781] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:52.140538 15600 raft_consensus.cc:482] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:52.140602 15600 raft_consensus.cc:3037] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.141731 15600 raft_consensus.cc:504] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.142135 15600 leader_election.cc:288] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 election: Requested vote from peers 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227), d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:52.142522 15499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0e82798fb745499c5f9de4b9a942c8" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fbb180f4204485ae19185fb8e59c96"
I20260808 08:19:52.142570 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0e82798fb745499c5f9de4b9a942c8" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3"
I20260808 08:19:52.142675 15285 raft_consensus.cc:3037] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.142663 15499 raft_consensus.cc:3037] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.144014 15499 raft_consensus.cc:2454] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 1.
I20260808 08:19:52.144014 15285 raft_consensus.cc:2454] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 1.
I20260808 08:19:52.144384 14964 leader_election.cc:302] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13b42f89788467ba9404259804c74d3, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:52.144572 15600 raft_consensus.cc:2781] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:52.144793 15600 raft_consensus.cc:686] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 LEADER]: Becoming Leader. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Running, Role: LEADER
I20260808 08:19:52.145149 15600 consensus_queue.cc:237] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } }
I20260808 08:19:52.148905 14831 catalog_manager.cc:5495] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1). New cstate: current_term: 1 leader_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } health_report { overall_health: UNKNOWN } } }
Aug 08 08:19:52 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:52.205921 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.196224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.12.13.62:35727)
0808 08:19:52.196399 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.196404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.196650 (+ 246us) server_negotiation.cc:407] Connection header received
0808 08:19:52.196799 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.196801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.196859 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.196942 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:52.197800 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.198423 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.199408 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.199614 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.199694 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.202411 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:52.202414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:52.202416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:52.202439 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:52.204015 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.204517 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.204521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.204522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.204609 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:52.204897 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:52.204901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:52.204902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:52.205089 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:52.205218 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.205522 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.205734 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":70}
I20260808 08:19:52.205921 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.196183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35727 (local address 127.0.0.1:46602)
0808 08:19:52.196574 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.196586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.196701 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.197023 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.197026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.197546 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:52.197654 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.197662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.198560 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.198563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.199277 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.199297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.199747 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.199750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.199819 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.200387 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:52.200400 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:52.202269 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:52.204155 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:52.204165 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:52.204171 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:52.204412 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:52.204709 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:52.204713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:52.204715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:52.204798 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:52.205219 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:52.205227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:52.205346 (+ 119us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.205472 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.205673 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":142,"threads_started":1}
I20260808 08:19:52.211655 14831 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:46602:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 08:19:52.212636 14831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:52.218770 15042 tablet_service.cc:1467] Processing CreateTablet for tablet d613cc9ec47a4449b4dbe8bf735daf17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2e91235e47949bb8bd30136a528c257]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:19:52.219100 15471 tablet_service.cc:1467] Processing CreateTablet for tablet d613cc9ec47a4449b4dbe8bf735daf17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2e91235e47949bb8bd30136a528c257]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:19:52.219660 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d613cc9ec47a4449b4dbe8bf735daf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.219812 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d613cc9ec47a4449b4dbe8bf735daf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.220584 15254 tablet_service.cc:1467] Processing CreateTablet for tablet d613cc9ec47a4449b4dbe8bf735daf17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2e91235e47949bb8bd30136a528c257]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:19:52.221310 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d613cc9ec47a4449b4dbe8bf735daf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.223125 15596 tablet_bootstrap.cc:492] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: Bootstrap starting.
I20260808 08:19:52.223129 15593 tablet_bootstrap.cc:492] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: Bootstrap starting.
I20260808 08:19:52.224213 15596 tablet_bootstrap.cc:654] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.224931 15593 tablet_bootstrap.cc:654] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.225054 15594 tablet_bootstrap.cc:492] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: Bootstrap starting.
I20260808 08:19:52.226001 15596 tablet_bootstrap.cc:492] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: No bootstrap required, opened a new log
I20260808 08:19:52.226111 15594 tablet_bootstrap.cc:654] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.226106 15596 ts_tablet_manager.cc:1397] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:19:52.226641 15593 tablet_bootstrap.cc:492] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: No bootstrap required, opened a new log
I20260808 08:19:52.226630 15596 raft_consensus.cc:348] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.226722 15593 ts_tablet_manager.cc:1397] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 08:19:52.226745 15596 raft_consensus.cc:374] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.226769 15596 raft_consensus.cc:729] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.226877 15596 consensus_queue.cc:260] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.227124 15596 ts_tablet_manager.cc:1428] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:52.227136 15601 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.227240 15593 raft_consensus.cc:348] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.227332 15601 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:52.227350 15593 raft_consensus.cc:374] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.227388 15593 raft_consensus.cc:729] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.227528 15593 consensus_queue.cc:260] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.227758 15593 ts_tablet_manager.cc:1428] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 08:19:52.227826 15600 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.227988 15600 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:52.228972 15594 tablet_bootstrap.cc:492] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: No bootstrap required, opened a new log
I20260808 08:19:52.229074 15594 ts_tablet_manager.cc:1397] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 08:19:52.229635 15594 raft_consensus.cc:348] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.229756 15594 raft_consensus.cc:374] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.229797 15594 raft_consensus.cc:729] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b42f89788467ba9404259804c74d3, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.229969 15594 consensus_queue.cc:260] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.230319 15594 ts_tablet_manager.cc:1428] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:52.230293 15602 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.230439 15602 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 08:19:52.260259 15146 tablet.cc:2369] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:19:52.261411 15560 tablet.cc:2369] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:19:52.266137 15355 tablet.cc:2369] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:19:52.346702 15602 raft_consensus.cc:482] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:52.346855 15602 raft_consensus.cc:504] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.348095 15602 leader_election.cc:288] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703), 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227)
I20260808 08:19:52.352327 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.348508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:57919 (local address 127.12.13.3:44227)
0808 08:19:52.348731 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.348737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.348833 (+ 96us) server_negotiation.cc:407] Connection header received
0808 08:19:52.349074 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.349078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.349129 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.349190 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.349536 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.350126 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.351230 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.351672 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.351760 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.351871 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.352070 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.352158 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260808 08:19:52.352340 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.348590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:34987 (local address 127.12.13.1:34703)
0808 08:19:52.348900 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.348905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.348921 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:19:52.349119 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.349122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.349190 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.349277 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:52.349584 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.350103 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.351310 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.351691 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.351834 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.351901 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.352125 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.352207 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260808 08:19:52.352350 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.348373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44227 (local address 127.12.13.2:57919)
0808 08:19:52.348731 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.348750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.348928 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.349238 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.349243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.349272 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.349411 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.349418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.350287 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.350290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.351094 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.351104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.351813 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.351817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.351922 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.351947 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.352019 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.352116 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260808 08:19:52.352397 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.348376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34703 (local address 127.12.13.2:34987)
0808 08:19:52.348817 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.348832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.348956 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.349309 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.349312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.349339 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:52.349441 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.349445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.350212 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.350218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.351143 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.351179 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.351860 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.351864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.351966 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.351985 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.352088 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.352159 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":195,"spinlock_wait_cycles":2432,"thread_start_us":54,"threads_started":1}
I20260808 08:19:52.352929 15499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613cc9ec47a4449b4dbe8bf735daf17" candidate_uuid: "d13b42f89788467ba9404259804c74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fbb180f4204485ae19185fb8e59c96" is_pre_election: true
I20260808 08:19:52.353107 15499 raft_consensus.cc:2454] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b42f89788467ba9404259804c74d3 in term 0.
I20260808 08:19:52.353117 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613cc9ec47a4449b4dbe8bf735daf17" candidate_uuid: "d13b42f89788467ba9404259804c74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" is_pre_election: true
I20260808 08:19:52.353363 15077 raft_consensus.cc:2454] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b42f89788467ba9404259804c74d3 in term 0.
I20260808 08:19:52.353892 15187 leader_election.cc:302] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, d13b42f89788467ba9404259804c74d3, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:52.354213 15602 raft_consensus.cc:2781] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:52.354285 15602 raft_consensus.cc:482] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:52.354313 15602 raft_consensus.cc:3037] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.355391 15602 raft_consensus.cc:504] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.355782 15602 leader_election.cc:288] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [CANDIDATE]: Term 1 election: Requested vote from peers fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703), 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227)
I20260808 08:19:52.356334 15499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613cc9ec47a4449b4dbe8bf735daf17" candidate_uuid: "d13b42f89788467ba9404259804c74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fbb180f4204485ae19185fb8e59c96"
I20260808 08:19:52.356338 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613cc9ec47a4449b4dbe8bf735daf17" candidate_uuid: "d13b42f89788467ba9404259804c74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8a11ed501048f6a61dd5c71e2d1657"
I20260808 08:19:52.356473 15077 raft_consensus.cc:3037] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.356473 15499 raft_consensus.cc:3037] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.357698 15077 raft_consensus.cc:2454] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b42f89788467ba9404259804c74d3 in term 1.
I20260808 08:19:52.357698 15499 raft_consensus.cc:2454] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b42f89788467ba9404259804c74d3 in term 1.
I20260808 08:19:52.358062 15187 leader_election.cc:302] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, d13b42f89788467ba9404259804c74d3, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:52.358251 15602 raft_consensus.cc:2781] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:52.358451 15602 raft_consensus.cc:686] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [term 1 LEADER]: Becoming Leader. State: Replica: d13b42f89788467ba9404259804c74d3, State: Running, Role: LEADER
I20260808 08:19:52.358860 15602 consensus_queue.cc:237] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.361366 15630 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: TxnStatusTablet state changed. Reason: New leader d13b42f89788467ba9404259804c74d3. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.361490 15630 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:19:52.361789 15636 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 08:19:52.362040 15636 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 08:19:52.362797 14831 catalog_manager.cc:5495] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b42f89788467ba9404259804c74d3 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } health_report { overall_health: UNKNOWN } } }
I20260808 08:19:52.386572 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.381949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45127 (local address 127.0.0.1:51452)
0808 08:19:52.382130 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.382148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.382266 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.382636 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.382638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.382645 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:52.382745 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.382750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.383794 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.383797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.384553 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.384564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.385147 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.385150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.385255 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.385323 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.386273 (+ 950us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.386291 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.386361 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.386446 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 08:19:52.386586 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.382102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.12.13.2:45127)
0808 08:19:52.382251 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.382256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.382274 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:19:52.382423 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.382426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.382510 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.382618 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:52.382864 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.383667 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.384674 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.385030 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.385136 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.385505 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.386165 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.386285 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.386375 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.386440 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 08:19:52.392364 15077 raft_consensus.cc:1261] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Refusing update from remote peer d13b42f89788467ba9404259804c74d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:52.392364 15499 raft_consensus.cc:1261] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Refusing update from remote peer d13b42f89788467ba9404259804c74d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:52.393157 15602 consensus_queue.cc:1035] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:52.393383 15630 consensus_queue.cc:1035] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:52.396690 15600 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: TxnStatusTablet state changed. Reason: New leader d13b42f89788467ba9404259804c74d3. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.396828 15600 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:52.398020 15601 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: TxnStatusTablet state changed. Reason: New leader d13b42f89788467ba9404259804c74d3. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.398156 15601 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:52.400600 15602 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.400724 15602 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:19:52.402482 15600 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.402597 15600 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P fc8a11ed501048f6a61dd5c71e2d1657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:52.404721 15642 mvcc.cc:205] Tried to move back new op lower bound from 7316181780031348736 to 7316181779904610304. Current Snapshot: MvccSnapshot[applied={T|T < 7316181780031348736 or (T in {7316181780031348736})}]
I20260808 08:19:52.406576 15630 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.406695 15630 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P d13b42f89788467ba9404259804c74d3: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:19:52.408227 15601 tablet_replica.cc:439] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d13b42f89788467ba9404259804c74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } } }
I20260808 08:19:52.408360 15601 tablet_replica.cc:442] T d613cc9ec47a4449b4dbe8bf735daf17 P 32fbb180f4204485ae19185fb8e59c96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:19:52.413070 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.408534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.12.13.1:34703)
0808 08:19:52.408658 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.408663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.408678 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:19:52.408762 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.408765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.408838 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.408969 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:52.409299 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.410108 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.411326 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.411700 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.411830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.412177 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.412639 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.412785 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.412856 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.412922 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 08:19:52.425792 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.421723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45127 (local address 127.12.13.1:57913)
0808 08:19:52.422089 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.422107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.422211 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.422487 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.422489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.422496 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:52.422611 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.422615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.423662 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.423665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.424380 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.424390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.424949 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.424951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.425023 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.425072 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.425464 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.425473 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.425564 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.425642 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":199,"threads_started":1}
I20260808 08:19:52.425781 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.421843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:57913 (local address 127.12.13.2:45127)
0808 08:19:52.422058 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.422063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.422189 (+ 126us) server_negotiation.cc:407] Connection header received
0808 08:19:52.422312 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.422316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.422375 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.422459 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:52.422712 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.423559 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.424496 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.424842 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.424949 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.425276 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.425370 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.425461 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.425576 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.425628 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 08:19:52.434059 15285 raft_consensus.cc:1261] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Refusing update from remote peer fc8a11ed501048f6a61dd5c71e2d1657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:52.434587 15499 raft_consensus.cc:1261] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Refusing update from remote peer fc8a11ed501048f6a61dd5c71e2d1657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:52.434707 15600 consensus_queue.cc:1035] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:52.435093 15608 consensus_queue.cc:1035] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:52.441299 15639 mvcc.cc:205] Tried to move back new op lower bound from 7316181780204171264 to 7316181779029979136. Current Snapshot: MvccSnapshot[applied={T|T < 7316181780204171264}]
I20260808 08:19:52.461460 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.457505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34703 (local address 127.12.13.2:41643)
0808 08:19:52.458125 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:52.458138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:52.458271 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:52.458530 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:52.458533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:52.458541 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:52.458675 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.458681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.459494 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.459499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.460135 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:52.460145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.460672 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.460676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:52.460777 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.460824 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.461149 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.461158 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 08:19:52.461226 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 08:19:52.461306 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":116,"threads_started":1}
I20260808 08:19:52.461447 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:52.457600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:41643 (local address 127.12.13.1:34703)
0808 08:19:52.457717 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:52.457722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:52.458219 (+ 497us) server_negotiation.cc:407] Connection header received
0808 08:19:52.458378 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:52.458380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:52.458434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:52.458519 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:52.458784 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.459357 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.460271 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:52.460577 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:52.460676 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:52.460977 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:52.461061 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:52.461146 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:52.461244 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 08:19:52.461297 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 08:19:52.493625 14831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:52.493884 14831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:52.496995 14831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Sending DeleteTablet for 3 replicas of tablet 3b0e82798fb745499c5f9de4b9a942c8
I20260808 08:19:52.497807 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0e82798fb745499c5f9de4b9a942c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:52 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:40158
I20260808 08:19:52.498350 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0e82798fb745499c5f9de4b9a942c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:52 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:59958
I20260808 08:19:52.498836 15254 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0e82798fb745499c5f9de4b9a942c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:52 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:51444
I20260808 08:19:52.498895 15670 tablet_replica.cc:330] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: stopping tablet replica
I20260808 08:19:52.499387 15670 raft_consensus.cc:2229] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:52.499764 15670 raft_consensus.cc:2258] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:52.500201 14831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:19:52.500646 14831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:52.501789 15670 ts_tablet_manager.cc:1905] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:52.505174 15670 ts_tablet_manager.cc:1918] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:52.505373 15670 log.cc:1196] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/3b0e82798fb745499c5f9de4b9a942c8
I20260808 08:19:52.505963 15042 tablet_service.cc:1467] Processing CreateTablet for tablet 1c30e3be298b4d75998557657bc269ed (DEFAULT_TABLE table=test-table [id=3d9812ebb00745508e245a5c28fe0a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:52.506287 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30e3be298b4d75998557657bc269ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.506461 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 1c30e3be298b4d75998557657bc269ed (DEFAULT_TABLE table=test-table [id=3d9812ebb00745508e245a5c28fe0a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:52.506728 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30e3be298b4d75998557657bc269ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.509181 15254 tablet_service.cc:1467] Processing CreateTablet for tablet 1c30e3be298b4d75998557657bc269ed (DEFAULT_TABLE table=test-table [id=3d9812ebb00745508e245a5c28fe0a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:52.509305 15593 tablet_bootstrap.cc:492] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Bootstrap starting.
I20260808 08:19:52.509475 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30e3be298b4d75998557657bc269ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:52.510401 15593 tablet_bootstrap.cc:654] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.511994 15594 tablet_bootstrap.cc:492] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Bootstrap starting.
I20260808 08:19:52.512009 15670 ts_tablet_manager.cc:1939] T 3b0e82798fb745499c5f9de4b9a942c8 P 32fbb180f4204485ae19185fb8e59c96: Deleting consensus metadata
I20260808 08:19:52.512358 15596 tablet_bootstrap.cc:492] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Bootstrap starting.
I20260808 08:19:52.513077 15594 tablet_bootstrap.cc:654] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.513492 15596 tablet_bootstrap.cc:654] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:52.513927 14807 catalog_manager.cc:4841] TS 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227): tablet 3b0e82798fb745499c5f9de4b9a942c8 (table test-table [id=b2c7be6420ef4143b0afd1174572f39e]) successfully deleted
I20260808 08:19:52.515703 15596 tablet_bootstrap.cc:492] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: No bootstrap required, opened a new log
I20260808 08:19:52.515797 15596 ts_tablet_manager.cc:1397] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:19:52.516248 15596 raft_consensus.cc:348] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.516371 15596 raft_consensus.cc:374] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.516398 15596 raft_consensus.cc:729] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.516513 15596 consensus_queue.cc:260] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.516780 15596 ts_tablet_manager.cc:1428] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:52.518278 15671 tablet_replica.cc:330] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: stopping tablet replica
I20260808 08:19:52.518594 15594 tablet_bootstrap.cc:492] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: No bootstrap required, opened a new log
I20260808 08:19:52.518642 15671 raft_consensus.cc:2229] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:52.518687 15594 ts_tablet_manager.cc:1397] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260808 08:19:52.519214 15594 raft_consensus.cc:348] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.519333 15594 raft_consensus.cc:374] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.519374 15594 raft_consensus.cc:729] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b42f89788467ba9404259804c74d3, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.519550 15594 consensus_queue.cc:260] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.520452 15594 ts_tablet_manager.cc:1428] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:19:52.520735 15671 raft_consensus.cc:2258] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:52.522742 15671 ts_tablet_manager.cc:1905] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:52.523329 15669 tablet_replica.cc:330] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: stopping tablet replica
I20260808 08:19:52.523653 15669 raft_consensus.cc:2229] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:19:52.524123 15669 raft_consensus.cc:2258] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:52.527284 15671 ts_tablet_manager.cc:1918] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:52.527428 15671 log.cc:1196] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/3b0e82798fb745499c5f9de4b9a942c8
I20260808 08:19:52.527763 15671 ts_tablet_manager.cc:1939] T 3b0e82798fb745499c5f9de4b9a942c8 P d13b42f89788467ba9404259804c74d3: Deleting consensus metadata
I20260808 08:19:52.528218 15669 ts_tablet_manager.cc:1905] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:52.529003 14804 catalog_manager.cc:4841] TS d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127): tablet 3b0e82798fb745499c5f9de4b9a942c8 (table test-table [id=b2c7be6420ef4143b0afd1174572f39e]) successfully deleted
I20260808 08:19:52.530121 15593 tablet_bootstrap.cc:492] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: No bootstrap required, opened a new log
I20260808 08:19:52.530227 15593 ts_tablet_manager.cc:1397] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20260808 08:19:52.530738 15593 raft_consensus.cc:348] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.530875 15593 raft_consensus.cc:374] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:52.530915 15593 raft_consensus.cc:729] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Initialized, Role: FOLLOWER
I20260808 08:19:52.531049 15593 consensus_queue.cc:260] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.531340 15593 ts_tablet_manager.cc:1428] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:52.534399 15669 ts_tablet_manager.cc:1918] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:52.534561 15669 log.cc:1196] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/3b0e82798fb745499c5f9de4b9a942c8
I20260808 08:19:52.534963 15669 ts_tablet_manager.cc:1939] T 3b0e82798fb745499c5f9de4b9a942c8 P fc8a11ed501048f6a61dd5c71e2d1657: Deleting consensus metadata
I20260808 08:19:52.536228 14806 catalog_manager.cc:4841] TS fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703): tablet 3b0e82798fb745499c5f9de4b9a942c8 (table test-table [id=b2c7be6420ef4143b0afd1174572f39e]) successfully deleted
I20260808 08:19:52.597783 15667 raft_consensus.cc:482] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:52.597998 15667 raft_consensus.cc:504] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.598582 15667 leader_election.cc:288] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127), 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227)
I20260808 08:19:52.599086 15498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30e3be298b4d75998557657bc269ed" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fbb180f4204485ae19185fb8e59c96" is_pre_election: true
I20260808 08:19:52.599078 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30e3be298b4d75998557657bc269ed" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3" is_pre_election: true
I20260808 08:19:52.599248 15498 raft_consensus.cc:2454] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 0.
I20260808 08:19:52.599248 15285 raft_consensus.cc:2454] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 0.
I20260808 08:19:52.599579 14967 leader_election.cc:302] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:52.599781 15667 raft_consensus.cc:2781] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:52.599850 15667 raft_consensus.cc:482] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:52.599890 15667 raft_consensus.cc:3037] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.601195 15667 raft_consensus.cc:504] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.601610 15667 leader_election.cc:288] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 election: Requested vote from peers d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127), 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227)
I20260808 08:19:52.601997 15498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30e3be298b4d75998557657bc269ed" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fbb180f4204485ae19185fb8e59c96"
I20260808 08:19:52.602005 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30e3be298b4d75998557657bc269ed" candidate_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3"
I20260808 08:19:52.602139 15498 raft_consensus.cc:3037] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.602139 15285 raft_consensus.cc:3037] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:52.603333 15285 raft_consensus.cc:2454] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 1.
I20260808 08:19:52.603339 15498 raft_consensus.cc:2454] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8a11ed501048f6a61dd5c71e2d1657 in term 1.
I20260808 08:19:52.603632 14967 leader_election.cc:302] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:52.603780 15667 raft_consensus.cc:2781] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:52.603878 15667 raft_consensus.cc:686] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 LEADER]: Becoming Leader. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Running, Role: LEADER
I20260808 08:19:52.604024 15667 consensus_queue.cc:237] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:52.605803 14834 catalog_manager.cc:5495] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1). New cstate: current_term: 1 leader_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } health_report { overall_health: UNKNOWN } } }
I20260808 08:19:52.639232 15498 raft_consensus.cc:1261] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Refusing update from remote peer fc8a11ed501048f6a61dd5c71e2d1657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:52.639393 15285 raft_consensus.cc:1261] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Refusing update from remote peer fc8a11ed501048f6a61dd5c71e2d1657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:52.639665 15667 consensus_queue.cc:1035] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:52.639858 15608 consensus_queue.cc:1035] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:52.643065 15640 mvcc.cc:205] Tried to move back new op lower bound from 7316181781046898688 to 7316181780907814912. Current Snapshot: MvccSnapshot[applied={T|T < 7316181781046898688}]
W20260808 08:19:52.658840 15642 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 08:19:54.229926 14831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:54.230088 14831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:54.232911 14831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Sending DeleteTablet for 3 replicas of tablet 1c30e3be298b4d75998557657bc269ed
I20260808 08:19:54.233652 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c30e3be298b4d75998557657bc269ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:54 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:40158
I20260808 08:19:54.233652 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c30e3be298b4d75998557657bc269ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:54 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:59958
I20260808 08:19:54.233737 15253 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c30e3be298b4d75998557657bc269ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:54 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:51444
I20260808 08:19:54.234198 15707 tablet_replica.cc:330] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: stopping tablet replica
I20260808 08:19:54.234349 15707 raft_consensus.cc:2229] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:54.234368 15709 tablet_replica.cc:330] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: stopping tablet replica
I20260808 08:19:54.234470 15707 raft_consensus.cc:2258] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:54.234530 15709 raft_consensus.cc:2229] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:54.234665 15709 raft_consensus.cc:2258] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:54.234782 15708 tablet_replica.cc:330] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: stopping tablet replica
I20260808 08:19:54.234935 15708 raft_consensus.cc:2229] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:19:54.235131 15708 raft_consensus.cc:2258] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:54.235543 15709 ts_tablet_manager.cc:1905] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:54.235761 15708 ts_tablet_manager.cc:1905] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:54.237847 15707 ts_tablet_manager.cc:1905] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:54.238444 15709 ts_tablet_manager.cc:1918] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 08:19:54.238562 15709 log.cc:1196] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/1c30e3be298b4d75998557657bc269ed
I20260808 08:19:54.238757 15708 ts_tablet_manager.cc:1918] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 08:19:54.238852 15708 log.cc:1196] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/1c30e3be298b4d75998557657bc269ed
I20260808 08:19:54.238901 15709 ts_tablet_manager.cc:1939] T 1c30e3be298b4d75998557657bc269ed P d13b42f89788467ba9404259804c74d3: Deleting consensus metadata
I20260808 08:19:54.239176 15708 ts_tablet_manager.cc:1939] T 1c30e3be298b4d75998557657bc269ed P fc8a11ed501048f6a61dd5c71e2d1657: Deleting consensus metadata
I20260808 08:19:54.240065 14804 catalog_manager.cc:4841] TS d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127): tablet 1c30e3be298b4d75998557657bc269ed (table test-table [id=3d9812ebb00745508e245a5c28fe0a0b]) successfully deleted
I20260808 08:19:54.240665 15707 ts_tablet_manager.cc:1918] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 08:19:54.240731 15707 log.cc:1196] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/1c30e3be298b4d75998557657bc269ed
I20260808 08:19:54.241083 15707 ts_tablet_manager.cc:1939] T 1c30e3be298b4d75998557657bc269ed P 32fbb180f4204485ae19185fb8e59c96: Deleting consensus metadata
I20260808 08:19:54.241647 14806 catalog_manager.cc:4841] TS fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703): tablet 1c30e3be298b4d75998557657bc269ed (table test-table [id=3d9812ebb00745508e245a5c28fe0a0b]) successfully deleted
I20260808 08:19:54.241917 14807 catalog_manager.cc:4841] TS 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227): tablet 1c30e3be298b4d75998557657bc269ed (table test-table [id=3d9812ebb00745508e245a5c28fe0a0b]) successfully deleted
Aug 08 08:19:54 dist-test-slave-6ntk krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 08:19:54 dist-test-slave-6ntk krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:54.267789 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.258094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.12.13.62:35727)
0808 08:19:54.258426 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:54.258436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:54.258453 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:19:54.258624 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:54.258628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:54.258689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:54.258798 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:54.259622 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.260202 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.261105 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.261349 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.261468 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.264293 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:54.264297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:54.264299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:54.264362 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:54.265983 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:54.266426 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:54.266431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:54.266433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:54.266524 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:54.266829 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:54.266832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:54.266834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:54.267038 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:54.267193 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:54.267464 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 08:19:54.267602 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260808 08:19:54.271322 14831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:19:54.271888 14831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:54.277092 15042 tablet_service.cc:1467] Processing CreateTablet for tablet d7fa34f2fc4f46ef9d0b1900635e1162 (DEFAULT_TABLE table=test-table [id=d0d920eb01c6434d938cf8b95d8fa4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:54.277217 15253 tablet_service.cc:1467] Processing CreateTablet for tablet d7fa34f2fc4f46ef9d0b1900635e1162 (DEFAULT_TABLE table=test-table [id=d0d920eb01c6434d938cf8b95d8fa4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:54.277462 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7fa34f2fc4f46ef9d0b1900635e1162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:54.277415 15471 tablet_service.cc:1467] Processing CreateTablet for tablet d7fa34f2fc4f46ef9d0b1900635e1162 (DEFAULT_TABLE table=test-table [id=d0d920eb01c6434d938cf8b95d8fa4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:54.277480 15253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7fa34f2fc4f46ef9d0b1900635e1162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:54.277670 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7fa34f2fc4f46ef9d0b1900635e1162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:54.280182 15727 tablet_bootstrap.cc:492] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Bootstrap starting.
I20260808 08:19:54.280562 15725 tablet_bootstrap.cc:492] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Bootstrap starting.
I20260808 08:19:54.280599 15726 tablet_bootstrap.cc:492] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Bootstrap starting.
I20260808 08:19:54.281412 15725 tablet_bootstrap.cc:654] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:54.281445 15727 tablet_bootstrap.cc:654] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:54.281478 15726 tablet_bootstrap.cc:654] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:54.282728 15725 tablet_bootstrap.cc:492] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: No bootstrap required, opened a new log
I20260808 08:19:54.282825 15725 ts_tablet_manager.cc:1397] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 08:19:54.282907 15727 tablet_bootstrap.cc:492] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: No bootstrap required, opened a new log
I20260808 08:19:54.282984 15726 tablet_bootstrap.cc:492] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: No bootstrap required, opened a new log
I20260808 08:19:54.282995 15727 ts_tablet_manager.cc:1397] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 08:19:54.283066 15726 ts_tablet_manager.cc:1397] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 08:19:54.283309 15725 raft_consensus.cc:348] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.283360 15727 raft_consensus.cc:348] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.283437 15725 raft_consensus.cc:374] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:54.283452 15727 raft_consensus.cc:374] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:54.283479 15725 raft_consensus.cc:729] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Initialized, Role: FOLLOWER
I20260808 08:19:54.283488 15727 raft_consensus.cc:729] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b42f89788467ba9404259804c74d3, State: Initialized, Role: FOLLOWER
I20260808 08:19:54.283541 15726 raft_consensus.cc:348] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.283624 15725 consensus_queue.cc:260] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.283638 15727 consensus_queue.cc:260] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.283730 15726 raft_consensus.cc:374] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:54.283777 15726 raft_consensus.cc:729] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Initialized, Role: FOLLOWER
I20260808 08:19:54.283948 15727 ts_tablet_manager.cc:1428] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 08:19:54.283905 15726 consensus_queue.cc:260] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.284013 15725 ts_tablet_manager.cc:1428] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 08:19:54.285100 15726 ts_tablet_manager.cc:1428] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 08:19:54.399246 15731 raft_consensus.cc:482] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:54.399441 15731 raft_consensus.cc:504] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.400625 15731 leader_election.cc:288] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703), d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:54.405457 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.401053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:53937 (local address 127.12.13.1:34703)
0808 08:19:54.401404 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:54.401418 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:54.401722 (+ 304us) server_negotiation.cc:407] Connection header received
0808 08:19:54.401940 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:54.401942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:54.402017 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:54.402108 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:54.402415 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.403022 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.404423 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.404840 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.404967 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.405034 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:54.405220 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 08:19:54.405307 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":162,"threads_started":1}
I20260808 08:19:54.405455 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.401055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:51319 (local address 127.12.13.2:45127)
0808 08:19:54.401699 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:54.401708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:54.401724 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:19:54.401784 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:54.401788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:54.401847 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:54.401981 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:19:54.402346 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.402952 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.404416 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.404827 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.404957 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.405029 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:54.405217 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 08:19:54.405293 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":98,"threads_started":1}
I20260808 08:19:54.405469 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.400919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45127 (local address 127.12.13.3:51319)
0808 08:19:54.401367 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:54.401382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:54.401497 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:54.402022 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:54.402025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:54.402046 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:54.402195 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.402203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.403072 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.403077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.404289 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.404299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.404959 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.404963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.405074 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.405103 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 08:19:54.405199 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 08:19:54.405279 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":80,"thread_start_us":147,"threads_started":1}
I20260808 08:19:54.405472 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.400919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34703 (local address 127.12.13.3:53937)
0808 08:19:54.401610 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:54.401624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:54.401750 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:54.402137 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:54.402139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:54.402156 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:19:54.402289 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.402297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.403161 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.403164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.404291 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.404302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.404971 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.404975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.405090 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.405109 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 08:19:54.405201 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 08:19:54.405284 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":348,"thread_start_us":155,"threads_started":1}
I20260808 08:19:54.405963 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7fa34f2fc4f46ef9d0b1900635e1162" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3" is_pre_election: true
I20260808 08:19:54.406106 15285 raft_consensus.cc:2454] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 0.
I20260808 08:19:54.406500 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7fa34f2fc4f46ef9d0b1900635e1162" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" is_pre_election: true
I20260808 08:19:54.406570 15397 leader_election.cc:302] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, d13b42f89788467ba9404259804c74d3; no voters:
I20260808 08:19:54.406677 15077 raft_consensus.cc:2454] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 0.
I20260808 08:19:54.406900 15731 raft_consensus.cc:2781] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:54.406975 15731 raft_consensus.cc:482] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:54.407040 15731 raft_consensus.cc:3037] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:54.408097 15731 raft_consensus.cc:504] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.408463 15731 leader_election.cc:288] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 election: Requested vote from peers fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703), d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:54.408901 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7fa34f2fc4f46ef9d0b1900635e1162" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3"
I20260808 08:19:54.408983 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7fa34f2fc4f46ef9d0b1900635e1162" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8a11ed501048f6a61dd5c71e2d1657"
I20260808 08:19:54.409041 15285 raft_consensus.cc:3037] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:54.409099 15077 raft_consensus.cc:3037] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:54.410349 15285 raft_consensus.cc:2454] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 1.
I20260808 08:19:54.410349 15077 raft_consensus.cc:2454] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 1.
I20260808 08:19:54.410702 15397 leader_election.cc:302] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, d13b42f89788467ba9404259804c74d3; no voters:
I20260808 08:19:54.411031 15731 raft_consensus.cc:2781] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:54.411283 15731 raft_consensus.cc:686] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 1 LEADER]: Becoming Leader. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Running, Role: LEADER
I20260808 08:19:54.411571 15731 consensus_queue.cc:237] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.414350 14830 catalog_manager.cc:5495] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3). New cstate: current_term: 1 leader_uuid: "32fbb180f4204485ae19185fb8e59c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } health_report { overall_health: HEALTHY } } }
I20260808 08:19:54.448390 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.444381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.12.13.3:44227)
0808 08:19:54.444668 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:54.444680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:54.444696 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:19:54.444778 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:54.444780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:54.444840 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:54.444982 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:54.445244 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.445795 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.446770 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.447017 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.447141 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.447448 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:54.447950 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:54.448095 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:54.448155 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 08:19:54.448213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260808 08:19:54.459678 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.455930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45127 (local address 127.12.13.3:51897)
0808 08:19:54.456281 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:54.456300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:54.456476 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:54.456758 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:54.456761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:54.456769 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:54.456916 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.456922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.457718 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.457721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.458394 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.458404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.458924 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.458927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.458993 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.459045 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:54.459417 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:54.459423 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:19:54.459477 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 08:19:54.459544 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260808 08:19:54.459667 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.456080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:51897 (local address 127.12.13.2:45127)
0808 08:19:54.456230 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:54.456235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:54.456415 (+ 180us) server_negotiation.cc:407] Connection header received
0808 08:19:54.456601 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:54.456604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:54.456658 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:54.456747 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:54.457058 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.457601 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.458546 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.458789 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.458890 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.459210 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:54.459289 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:54.459361 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:54.459511 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 08:19:54.459561 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 08:19:54.467013 15077 raft_consensus.cc:1261] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Refusing update from remote peer 32fbb180f4204485ae19185fb8e59c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:54.467576 15738 consensus_queue.cc:1035] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:54.469537 15285 raft_consensus.cc:1261] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Refusing update from remote peer 32fbb180f4204485ae19185fb8e59c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:54.470041 15738 consensus_queue.cc:1035] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:54.498677 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.491943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:34717 (local address 127.12.13.3:44227)
0808 08:19:54.492076 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:54.492083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:54.492266 (+ 183us) server_negotiation.cc:407] Connection header received
0808 08:19:54.492451 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:54.492456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:54.492528 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:54.492617 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:19:54.492955 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.493499 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.495417 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.495818 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.495949 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.498043 (+ 2094us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:19:54.498171 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:19:54.498258 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:54.498437 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 08:19:54.498514 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 08:19:54.498749 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:54.491802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44227 (local address 127.12.13.2:34717)
0808 08:19:54.492145 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:54.492173 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:54.492321 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:54.492628 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:54.492630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:54.492637 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:19:54.492747 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.492780 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.493611 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.493614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.495298 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:54.495308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:54.495959 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:54.495962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:54.496081 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:54.496133 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:19:54.498303 (+ 2170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:19:54.498313 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 08:19:54.498403 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 08:19:54.498525 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260808 08:19:54.539505 14830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:54.539674 14830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:54.542439 14830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Sending DeleteTablet for 3 replicas of tablet d7fa34f2fc4f46ef9d0b1900635e1162
I20260808 08:19:54.543097 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet d7fa34f2fc4f46ef9d0b1900635e1162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:54 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:40158
I20260808 08:19:54.543273 15253 tablet_service.cc:1514] Processing DeleteTablet for tablet d7fa34f2fc4f46ef9d0b1900635e1162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:54 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:51444
I20260808 08:19:54.543342 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet d7fa34f2fc4f46ef9d0b1900635e1162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:54 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:59958
I20260808 08:19:54.543398 15708 tablet_replica.cc:330] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: stopping tablet replica
I20260808 08:19:54.543521 15707 tablet_replica.cc:330] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: stopping tablet replica
I20260808 08:19:54.543568 15708 raft_consensus.cc:2229] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:54.543648 15707 raft_consensus.cc:2229] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:19:54.543725 15708 raft_consensus.cc:2258] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:54.543944 15709 tablet_replica.cc:330] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: stopping tablet replica
I20260808 08:19:54.543992 15707 raft_consensus.cc:2258] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:54.544100 15709 raft_consensus.cc:2229] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:54.544262 15709 raft_consensus.cc:2258] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:54.544922 15707 ts_tablet_manager.cc:1905] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:54.544988 15708 ts_tablet_manager.cc:1905] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:54.544917 14830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:19:54.545393 14830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:19:54.547396 15709 ts_tablet_manager.cc:1905] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:54.547930 15708 ts_tablet_manager.cc:1918] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:54.548007 15708 log.cc:1196] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/d7fa34f2fc4f46ef9d0b1900635e1162
I20260808 08:19:54.548280 15708 ts_tablet_manager.cc:1939] T d7fa34f2fc4f46ef9d0b1900635e1162 P fc8a11ed501048f6a61dd5c71e2d1657: Deleting consensus metadata
I20260808 08:19:54.549126 15707 ts_tablet_manager.cc:1918] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:54.549222 15707 log.cc:1196] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/d7fa34f2fc4f46ef9d0b1900635e1162
I20260808 08:19:54.549414 14806 catalog_manager.cc:4841] TS fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703): tablet d7fa34f2fc4f46ef9d0b1900635e1162 (table test-table [id=d0d920eb01c6434d938cf8b95d8fa4be]) successfully deleted
I20260808 08:19:54.549495 15707 ts_tablet_manager.cc:1939] T d7fa34f2fc4f46ef9d0b1900635e1162 P 32fbb180f4204485ae19185fb8e59c96: Deleting consensus metadata
I20260808 08:19:54.550189 14807 catalog_manager.cc:4841] TS 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227): tablet d7fa34f2fc4f46ef9d0b1900635e1162 (table test-table [id=d0d920eb01c6434d938cf8b95d8fa4be]) successfully deleted
I20260808 08:19:54.550233 15709 ts_tablet_manager.cc:1918] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:19:54.550313 15709 log.cc:1196] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/d7fa34f2fc4f46ef9d0b1900635e1162
I20260808 08:19:54.550585 15709 ts_tablet_manager.cc:1939] T d7fa34f2fc4f46ef9d0b1900635e1162 P d13b42f89788467ba9404259804c74d3: Deleting consensus metadata
I20260808 08:19:54.551330 14804 catalog_manager.cc:4841] TS d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127): tablet d7fa34f2fc4f46ef9d0b1900635e1162 (table test-table [id=d0d920eb01c6434d938cf8b95d8fa4be]) successfully deleted
I20260808 08:19:54.553781 15042 tablet_service.cc:1467] Processing CreateTablet for tablet c275fb9481da45c79ad6752213db1c29 (DEFAULT_TABLE table=test-table [id=572cc9b82b394b3db090f8b4bc979ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:54.553943 15253 tablet_service.cc:1467] Processing CreateTablet for tablet c275fb9481da45c79ad6752213db1c29 (DEFAULT_TABLE table=test-table [id=572cc9b82b394b3db090f8b4bc979ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:54.554167 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c275fb9481da45c79ad6752213db1c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:54.554122 15471 tablet_service.cc:1467] Processing CreateTablet for tablet c275fb9481da45c79ad6752213db1c29 (DEFAULT_TABLE table=test-table [id=572cc9b82b394b3db090f8b4bc979ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:19:54.554267 15253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c275fb9481da45c79ad6752213db1c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:54.554381 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c275fb9481da45c79ad6752213db1c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:54.556764 15726 tablet_bootstrap.cc:492] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657: Bootstrap starting.
I20260808 08:19:54.556811 15727 tablet_bootstrap.cc:492] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Bootstrap starting.
I20260808 08:19:54.557660 15726 tablet_bootstrap.cc:654] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:54.557688 15727 tablet_bootstrap.cc:654] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:54.559085 15726 tablet_bootstrap.cc:492] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657: No bootstrap required, opened a new log
I20260808 08:19:54.559180 15726 ts_tablet_manager.cc:1397] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 08:19:54.559703 15726 raft_consensus.cc:348] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.559857 15726 raft_consensus.cc:374] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:54.559901 15726 raft_consensus.cc:729] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a11ed501048f6a61dd5c71e2d1657, State: Initialized, Role: FOLLOWER
I20260808 08:19:54.560068 15726 consensus_queue.cc:260] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.560436 15726 ts_tablet_manager.cc:1428] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 08:19:54.562101 15725 tablet_bootstrap.cc:492] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Bootstrap starting.
I20260808 08:19:54.563066 15725 tablet_bootstrap.cc:654] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:54.564638 15725 tablet_bootstrap.cc:492] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: No bootstrap required, opened a new log
I20260808 08:19:54.564723 15725 ts_tablet_manager.cc:1397] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 08:19:54.565158 15725 raft_consensus.cc:348] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.565260 15725 raft_consensus.cc:374] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:54.565284 15725 raft_consensus.cc:729] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Initialized, Role: FOLLOWER
I20260808 08:19:54.565464 15725 consensus_queue.cc:260] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.565752 15725 ts_tablet_manager.cc:1428] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 08:19:54.566781 15731 raft_consensus.cc:482] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:19:54.566877 15731 raft_consensus.cc:504] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.567281 15731 leader_election.cc:288] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703), d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:54.567574 15727 tablet_bootstrap.cc:492] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: No bootstrap required, opened a new log
I20260808 08:19:54.567677 15727 ts_tablet_manager.cc:1397] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260808 08:19:54.568079 15727 raft_consensus.cc:348] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.568187 15727 raft_consensus.cc:374] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:54.568219 15727 raft_consensus.cc:729] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b42f89788467ba9404259804c74d3, State: Initialized, Role: FOLLOWER
I20260808 08:19:54.568337 15727 consensus_queue.cc:260] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.568586 15727 ts_tablet_manager.cc:1428] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:19:54.568799 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c275fb9481da45c79ad6752213db1c29" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" is_pre_election: true
I20260808 08:19:54.568926 15076 raft_consensus.cc:2454] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 0.
I20260808 08:19:54.569262 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c275fb9481da45c79ad6752213db1c29" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3" is_pre_election: true
I20260808 08:19:54.569383 15285 raft_consensus.cc:2454] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 0.
I20260808 08:19:54.569705 15398 leader_election.cc:302] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, fc8a11ed501048f6a61dd5c71e2d1657; no voters:
I20260808 08:19:54.569952 15731 raft_consensus.cc:2781] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:19:54.570020 15731 raft_consensus.cc:482] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:19:54.570066 15731 raft_consensus.cc:3037] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:54.571385 15731 raft_consensus.cc:504] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.571789 15731 leader_election.cc:288] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 election: Requested vote from peers fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703), d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127)
I20260808 08:19:54.572129 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c275fb9481da45c79ad6752213db1c29" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8a11ed501048f6a61dd5c71e2d1657"
I20260808 08:19:54.572290 15076 raft_consensus.cc:3037] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:54.572283 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c275fb9481da45c79ad6752213db1c29" candidate_uuid: "32fbb180f4204485ae19185fb8e59c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b42f89788467ba9404259804c74d3"
I20260808 08:19:54.572453 15285 raft_consensus.cc:3037] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:54.573596 15076 raft_consensus.cc:2454] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 1.
I20260808 08:19:54.573629 15285 raft_consensus.cc:2454] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32fbb180f4204485ae19185fb8e59c96 in term 1.
I20260808 08:19:54.573982 15397 leader_election.cc:302] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fbb180f4204485ae19185fb8e59c96, d13b42f89788467ba9404259804c74d3; no voters:
I20260808 08:19:54.574358 15731 raft_consensus.cc:2781] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:54.574483 15731 raft_consensus.cc:686] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 1 LEADER]: Becoming Leader. State: Replica: 32fbb180f4204485ae19185fb8e59c96, State: Running, Role: LEADER
I20260808 08:19:54.574637 15731 consensus_queue.cc:237] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } }
I20260808 08:19:54.576433 14830 catalog_manager.cc:5495] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3). New cstate: current_term: 1 leader_uuid: "32fbb180f4204485ae19185fb8e59c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32fbb180f4204485ae19185fb8e59c96" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44227 } health_report { overall_health: HEALTHY } } }
I20260808 08:19:54.595495 15076 raft_consensus.cc:1261] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Refusing update from remote peer 32fbb180f4204485ae19185fb8e59c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:54.595588 15285 raft_consensus.cc:1261] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Refusing update from remote peer 32fbb180f4204485ae19185fb8e59c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:19:54.596019 15731 consensus_queue.cc:1035] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b42f89788467ba9404259804c74d3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:19:54.596254 15738 consensus_queue.cc:1035] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8a11ed501048f6a61dd5c71e2d1657" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 08:19:54.612600 15686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 08:19:56.175441 14830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:56.175606 14830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:19:56.178341 14830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 141d2298af304f79b68e641cc96d1460: Sending DeleteTablet for 3 replicas of tablet c275fb9481da45c79ad6752213db1c29
I20260808 08:19:56.178983 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet c275fb9481da45c79ad6752213db1c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:56 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:40158
I20260808 08:19:56.179102 15254 tablet_service.cc:1514] Processing DeleteTablet for tablet c275fb9481da45c79ad6752213db1c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:56 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:51444
I20260808 08:19:56.179176 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet c275fb9481da45c79ad6752213db1c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:19:56 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:59958
I20260808 08:19:56.179466 15811 tablet_replica.cc:330] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657: stopping tablet replica
I20260808 08:19:56.179646 15811 raft_consensus.cc:2229] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:56.179790 15811 raft_consensus.cc:2258] T c275fb9481da45c79ad6752213db1c29 P fc8a11ed501048f6a61dd5c71e2d1657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:56.179909 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 14939
I20260808 08:19:56.190109 15812 tablet_replica.cc:330] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: stopping tablet replica
I20260808 08:19:56.190327 15812 raft_consensus.cc:2229] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:19:56.190526 15812 raft_consensus.cc:2258] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:56.192559 15812 ts_tablet_manager.cc:1905] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:56.196406 15812 ts_tablet_manager.cc:1918] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 08:19:56.196528 15812 log.cc:1196] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/c275fb9481da45c79ad6752213db1c29
I20260808 08:19:56.196816 15812 ts_tablet_manager.cc:1939] T c275fb9481da45c79ad6752213db1c29 P d13b42f89788467ba9404259804c74d3: Deleting consensus metadata
I20260808 08:19:56.197809 14804 catalog_manager.cc:4841] TS d13b42f89788467ba9404259804c74d3 (127.12.13.2:45127): tablet c275fb9481da45c79ad6752213db1c29 (table test-table [id=572cc9b82b394b3db090f8b4bc979ae6]) successfully deleted
I20260808 08:19:56.200776 15813 tablet_replica.cc:330] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: stopping tablet replica
W20260808 08:19:56.200759 14806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:34703 (error 108)
I20260808 08:19:56.200935 15813 raft_consensus.cc:2229] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:19:56.201208 15813 raft_consensus.cc:2258] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:19:56.201552 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 15157
I20260808 08:19:56.202713 15813 ts_tablet_manager.cc:1905] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:19:56.206390 15813 ts_tablet_manager.cc:1918] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 08:19:56.206471 15813 log.cc:1196] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/c275fb9481da45c79ad6752213db1c29
I20260808 08:19:56.206763 15813 ts_tablet_manager.cc:1939] T c275fb9481da45c79ad6752213db1c29 P 32fbb180f4204485ae19185fb8e59c96: Deleting consensus metadata
I20260808 08:19:56.208729 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:56.201631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34703 (local address 127.0.0.1:40180)
0808 08:19:56.208459 (+ 6828us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:56.208586 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6634,"thread_start_us":86,"threads_started":1}
I20260808 08:19:56.208783 14807 catalog_manager.cc:4841] TS 32fbb180f4204485ae19185fb8e59c96 (127.12.13.3:44227): tablet c275fb9481da45c79ad6752213db1c29 (table test-table [id=572cc9b82b394b3db090f8b4bc979ae6]) successfully deleted
W20260808 08:19:56.208886 14806 catalog_manager.cc:4584] TS fc8a11ed501048f6a61dd5c71e2d1657 (127.12.13.1:34703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c275fb9481da45c79ad6752213db1c29: Network error: Client connection negotiation failed: client connection to 127.12.13.1:34703: connect: Connection refused (error 111)
I20260808 08:19:56.216310 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 15369
I20260808 08:19:56.229665 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 14780
2026-08-08T08:19:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6684 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:19:56 dist-test-slave-6ntk krb5kdc[15821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:19:56 dist-test-slave-6ntk krb5kdc[15821](info): set up 2 sockets
Aug 08 08:19:56 dist-test-slave-6ntk krb5kdc[15821](info): commencing operation
krb5kdc: starting...
W20260808 08:19:57.522529 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:19:57 dist-test-slave-6ntk krb5kdc[15821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:19:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:19:57.665808 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:41353
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:41353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:57.790287 15845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:57.790661 15845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:57.790773 15845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:57.795023 15845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:19:57.795146 15845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:57.795197 15845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:57.795258 15845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:19:57.795295 15845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:19:57.800915 15845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:41353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:41353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:57.802193 15845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:57.803324 15845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:57.812989 15854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:57.813009 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:57.813017 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:57.813099 15845 server_base.cc:1034] running on GCE node
I20260808 08:19:57.814110 15845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:57.815388 15845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:57.816562 15845 hybrid_clock.cc:648] HybridClock initialized: now 1786177197816541 us; error 45 us; skew 500 ppm
Aug 08 08:19:57 dist-test-slave-6ntk krb5kdc[15821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:57.819788 15845 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:19:57.820902 15845 webserver.cc:460] Webserver started at http://127.12.13.62:37247/ using document root <none> and password file <none>
I20260808 08:19:57.821561 15845 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:57.821661 15845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:57.821998 15845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:57.824249 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "505e2a822225406b9c95d7a94d363971"
format_stamp: "Formatted at 2026-08-08 08:19:57 on dist-test-slave-6ntk"
server_key: "abe521a977b5d4c7d75ffc6e5a86bfc0"
server_key_iv: "217841bd1cf6858481734767b0ceaab1"
server_key_version: "encryptionkey@0"
I20260808 08:19:57.824892 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "505e2a822225406b9c95d7a94d363971"
format_stamp: "Formatted at 2026-08-08 08:19:57 on dist-test-slave-6ntk"
server_key: "abe521a977b5d4c7d75ffc6e5a86bfc0"
server_key_iv: "217841bd1cf6858481734767b0ceaab1"
server_key_version: "encryptionkey@0"
I20260808 08:19:57.829072 15845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 08:19:57.831805 15867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:57.833055 15845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:19:57.833205 15845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "505e2a822225406b9c95d7a94d363971"
format_stamp: "Formatted at 2026-08-08 08:19:57 on dist-test-slave-6ntk"
server_key: "abe521a977b5d4c7d75ffc6e5a86bfc0"
server_key_iv: "217841bd1cf6858481734767b0ceaab1"
server_key_version: "encryptionkey@0"
I20260808 08:19:57.833328 15845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:57.847771 15845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:57.851002 15845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:57.851239 15845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:57.859006 15845 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:41353
I20260808 08:19:57.859036 15946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:41353 every 8 connection(s)
I20260808 08:19:57.860044 15845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:19:57.863152 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:19:57.864203 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 15845
I20260808 08:19:57.864339 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:19:57.864667 12340 external_mini_cluster.cc:1428] Setting key 81cf0b835d9ffeedfd75d64470ac95ea
I20260808 08:19:57.869922 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971: Bootstrap starting.
I20260808 08:19:57.872419 15949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971: Neither blocks nor log segments found. Creating new log.
I20260808 08:19:57.873224 15949 log.cc:824] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:19:57 dist-test-slave-6ntk krb5kdc[15821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:57.875577 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971: No bootstrap required, opened a new log
I20260808 08:19:57.879496 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:57.866322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.12.13.62:41353)
0808 08:19:57.866814 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:57.866864 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:57.866900 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:19:57.867538 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:57.867554 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:57.867903 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:57.868259 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:57.869136 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:57.870263 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:57.871133 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:57.871608 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:57.871800 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:57.874705 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:57.874711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:57.874725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:57.874753 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:57.876509 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:57.876993 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:57.876998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:57.877003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:57.877118 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:57.877418 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:57.877422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:57.877423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:57.877852 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:57.878111 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:57.878412 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 08:19:57.878723 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260808 08:19:57.879376 15949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505e2a822225406b9c95d7a94d363971" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41353 } }
I20260808 08:19:57.879652 15949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:19:57.879715 15949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505e2a822225406b9c95d7a94d363971, State: Initialized, Role: FOLLOWER
I20260808 08:19:57.880198 15949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505e2a822225406b9c95d7a94d363971" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41353 } }
I20260808 08:19:57.880381 15949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:19:57.880453 15949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:19:57.880605 15949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:19:57.881939 15949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505e2a822225406b9c95d7a94d363971" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41353 } }
I20260808 08:19:57.882344 15949 leader_election.cc:302] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 505e2a822225406b9c95d7a94d363971; no voters:
I20260808 08:19:57.882692 15949 leader_election.cc:288] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:19:57.882819 15958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:19:57.883098 15958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [term 1 LEADER]: Becoming Leader. State: Replica: 505e2a822225406b9c95d7a94d363971, State: Running, Role: LEADER
I20260808 08:19:57.883529 15958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505e2a822225406b9c95d7a94d363971" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41353 } }
I20260808 08:19:57.883996 15949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:19:57.885380 15960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 505e2a822225406b9c95d7a94d363971. Latest consensus state: current_term: 1 leader_uuid: "505e2a822225406b9c95d7a94d363971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505e2a822225406b9c95d7a94d363971" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41353 } } }
I20260808 08:19:57.885411 15959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "505e2a822225406b9c95d7a94d363971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505e2a822225406b9c95d7a94d363971" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41353 } } }
I20260808 08:19:57.885528 15960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:57.885526 15959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971 [sys.catalog]: This master's current role is: LEADER
I20260808 08:19:57.885957 15969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:19:57.889285 15969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:19:57.895247 15969 catalog_manager.cc:1347] Generated new cluster ID: 5cf4ed3582024d8f88cf4b9ec835e911
I20260808 08:19:57.895335 15969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:19:57.901007 15969 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:19:57.902113 15969 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:19:57.908495 15969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 505e2a822225406b9c95d7a94d363971: Generated new TSK 0
I20260808 08:19:57.909372 15969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:19:57.953751 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:41353
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:58.075575 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:58.075927 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:58.076007 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:58.080188 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:58.080291 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:58.080389 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:19:58.086223 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:41353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:58.087399 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:58.088506 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:58.098748 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:58.098748 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:58.098853 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:58.099164 15992 server_base.cc:1034] running on GCE node
I20260808 08:19:58.099673 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:58.100455 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:58.101696 15992 hybrid_clock.cc:648] HybridClock initialized: now 1786177198101676 us; error 31 us; skew 500 ppm
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[15821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:58.105123 15992 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:19:58.106308 15992 webserver.cc:460] Webserver started at http://127.12.13.1:44501/ using document root <none> and password file <none>
I20260808 08:19:58.106999 15992 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:58.107110 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:58.107376 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:58.109750 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "9dcf13f3f98749a8a391715d4ecde873"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "45d68c41196b43bd202d84cc43b3abf2"
server_key_iv: "e9fcfb048aba3620ebc1c48437a33a62"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.110455 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "9dcf13f3f98749a8a391715d4ecde873"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "45d68c41196b43bd202d84cc43b3abf2"
server_key_iv: "e9fcfb048aba3620ebc1c48437a33a62"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.114697 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:19:58.117436 16014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.118968 15992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:19:58.119139 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "9dcf13f3f98749a8a391715d4ecde873"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "45d68c41196b43bd202d84cc43b3abf2"
server_key_iv: "e9fcfb048aba3620ebc1c48437a33a62"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.119275 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:58.158929 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:58.162506 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:58.162760 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:58.163471 15992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:58.164490 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:58.164572 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.164659 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:58.164721 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.174996 15992 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:38393
I20260808 08:19:58.175028 16183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:38393 every 8 connection(s)
I20260808 08:19:58.176018 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:19:58.182667 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 15992
I20260808 08:19:58.182798 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:19:58.183125 12340 external_mini_cluster.cc:1428] Setting key 6ffca66b334169970a07aee6699981d8
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[15821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:58.194159 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:58.178002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:42083 (local address 127.12.13.62:41353)
0808 08:19:58.178145 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:58.178149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:58.179328 (+ 1179us) server_negotiation.cc:407] Connection header received
0808 08:19:58.180215 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:58.180217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:58.180272 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:58.180403 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:58.181729 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.182408 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.184022 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.184371 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.184498 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:58.188226 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:58.188232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:58.188234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:58.188296 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:58.190528 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:58.191845 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:58.191849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:58.191851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:58.191960 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:58.192490 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:58.192493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:58.192495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:58.192773 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:58.192888 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:58.193790 (+ 902us) server_negotiation.cc:299] Negotiation successful
0808 08:19:58.193979 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:19:58.195188 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:58.178442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:41353 (local address 127.12.13.1:42083)
0808 08:19:58.179155 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:58.179201 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:58.180015 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:58.180558 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:58.180565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:58.180958 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:58.181570 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:58.181586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.182554 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.182557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:58.183902 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:58.183914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.184579 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.184584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:58.184802 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:58.185641 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:58.185666 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:58.187952 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:58.190664 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:58.190670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:58.190690 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:58.191053 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:58.192096 (+ 1043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:58.192099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:58.192100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:58.192353 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:58.193005 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:58.193014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:58.193471 (+ 457us) client_negotiation.cc:769] Sending connection context
0808 08:19:58.193744 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 08:19:58.194157 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":116,"threads_started":1}
I20260808 08:19:58.197511 16184 heartbeater.cc:347] Connected to a master server at 127.12.13.62:41353
I20260808 08:19:58.197801 16184 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:19:58.198406 16184 heartbeater.cc:511] Master 127.12.13.62:41353 requested a full tablet report, sending...
I20260808 08:19:58.200223 15892 ts_manager.cc:194] Registered new tserver with Master: 9dcf13f3f98749a8a391715d4ecde873 (127.12.13.1:38393)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:19:58.201951 15892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:42083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:19:58.221469 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:41353
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:58.343015 16192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:58.343319 16192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:58.343364 16192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:58.347649 16192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:58.347754 16192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:58.347846 16192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:19:58.353415 16192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:41353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:58.354648 16192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:58.355755 16192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:58.366046 16203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:58.366021 16198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:58.366029 16197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:58.366442 16192 server_base.cc:1034] running on GCE node
I20260808 08:19:58.366907 16192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:58.367595 16192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:58.368822 16192 hybrid_clock.cc:648] HybridClock initialized: now 1786177198368782 us; error 50 us; skew 500 ppm
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[15821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:58.372004 16192 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:19:58.372947 16192 webserver.cc:460] Webserver started at http://127.12.13.2:37965/ using document root <none> and password file <none>
I20260808 08:19:58.373540 16192 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:58.373608 16192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:58.373796 16192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:58.376161 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "6a19491d8418403fa87f5a4e984120db"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "8b4cbb5d0c88d3e2179d51753fc4dd41"
server_key_iv: "9af71fc589e95118637d56a83937145d"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.376722 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "6a19491d8418403fa87f5a4e984120db"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "8b4cbb5d0c88d3e2179d51753fc4dd41"
server_key_iv: "9af71fc589e95118637d56a83937145d"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.380954 16192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:19:58.383960 16214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.385425 16192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:19:58.385538 16192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "6a19491d8418403fa87f5a4e984120db"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "8b4cbb5d0c88d3e2179d51753fc4dd41"
server_key_iv: "9af71fc589e95118637d56a83937145d"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.385669 16192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:58.405424 16192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:58.408861 16192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:58.409036 16192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:58.409648 16192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:58.410607 16192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:58.410671 16192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.410717 16192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:58.410733 16192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.420846 16192 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:37987
I20260808 08:19:58.420874 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:37987 every 8 connection(s)
I20260808 08:19:58.421993 16192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:19:58.430469 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 16192
I20260808 08:19:58.430565 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:19:58.430868 12340 external_mini_cluster.cc:1428] Setting key a166917726a2f9c83db77b5f15eef76b
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[15821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:58.436964 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:58.424139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:55497 (local address 127.12.13.62:41353)
0808 08:19:58.424335 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:58.424341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:58.425610 (+ 1269us) server_negotiation.cc:407] Connection header received
0808 08:19:58.426605 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:58.426609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:58.426659 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:58.426742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:58.427945 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.428490 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.429604 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.429812 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.429973 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:58.432666 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:58.432673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:58.432679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:58.432710 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:58.434544 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:58.435251 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:58.435254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:58.435255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:58.435342 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:58.435749 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:58.435751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:58.435752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:58.435928 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:58.436005 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:58.436715 (+ 710us) server_negotiation.cc:299] Negotiation successful
0808 08:19:58.436838 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:19:58.438328 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:58.424661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:41353 (local address 127.12.13.2:55497)
0808 08:19:58.425432 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:58.425479 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:58.426371 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:58.426918 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:58.426926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:58.427302 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:58.427812 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:58.427826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.428607 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.428610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:58.429494 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:58.429503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.429995 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.429997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:58.430139 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:58.430833 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:58.430849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:58.432497 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:58.434687 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:58.434696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:58.434714 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:58.435140 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:58.435441 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:58.435445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:58.435448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:58.435649 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:58.436042 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:58.436052 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:58.436427 (+ 375us) client_negotiation.cc:769] Sending connection context
0808 08:19:58.436661 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 08:19:58.437051 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":132,"threads_started":1}
I20260808 08:19:58.439647 16397 heartbeater.cc:347] Connected to a master server at 127.12.13.62:41353
I20260808 08:19:58.439921 16397 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:58.440452 16397 heartbeater.cc:511] Master 127.12.13.62:41353 requested a full tablet report, sending...
I20260808 08:19:58.441571 15892 ts_manager.cc:194] Registered new tserver with Master: 6a19491d8418403fa87f5a4e984120db (127.12.13.2:37987)
I20260808 08:19:58.442224 15892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:55497
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:19:58.465210 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:41353
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:19:58.589824 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:19:58.590181 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:19:58.590265 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:19:58.594333 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:19:58.594427 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:19:58.594523 16402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:19:58.599834 16402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:41353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:19:58.600997 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:19:58.602161 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:19:58.612751 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:58.612742 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:19:58.612737 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:19:58.612835 16402 server_base.cc:1034] running on GCE node
I20260808 08:19:58.613524 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:19:58.614199 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:19:58.615389 16402 hybrid_clock.cc:648] HybridClock initialized: now 1786177198615352 us; error 49 us; skew 500 ppm
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[15821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:19:58.618429 16402 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:19:58.619437 16402 webserver.cc:460] Webserver started at http://127.12.13.3:33067/ using document root <none> and password file <none>
I20260808 08:19:58.620019 16402 fs_manager.cc:359] Metadata directory not provided
I20260808 08:19:58.620082 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:19:58.620260 16402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:19:58.622859 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "0d648fe08a1c4c718adf25814c0fc2c3"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "495d3156c893e136356637a604e78996"
server_key_iv: "e79ece689b186a65f57670baa4b82015"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.623663 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "0d648fe08a1c4c718adf25814c0fc2c3"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "495d3156c893e136356637a604e78996"
server_key_iv: "e79ece689b186a65f57670baa4b82015"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.628060 16402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:19:58.630842 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.632171 16402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:19:58.632319 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "0d648fe08a1c4c718adf25814c0fc2c3"
format_stamp: "Formatted at 2026-08-08 08:19:58 on dist-test-slave-6ntk"
server_key: "495d3156c893e136356637a604e78996"
server_key_iv: "e79ece689b186a65f57670baa4b82015"
server_key_version: "encryptionkey@0"
I20260808 08:19:58.632460 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:19:58.649645 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:19:58.652688 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:19:58.652915 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:19:58.653570 16402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:19:58.654533 16402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:19:58.654582 16402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.654624 16402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:19:58.654673 16402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:19:58.664151 16402 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:40207
I20260808 08:19:58.664209 16590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:40207 every 8 connection(s)
I20260808 08:19:58.665114 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:19:58.674026 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 16402
I20260808 08:19:58.674177 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:19:58.674487 12340 external_mini_cluster.cc:1428] Setting key 63771b7ce2b9cb1c1f4c1d8c2ecda3bc
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[15821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:19:58.679648 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:58.667232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:53021 (local address 127.12.13.62:41353)
0808 08:19:58.667375 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:58.667379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:58.668471 (+ 1092us) server_negotiation.cc:407] Connection header received
0808 08:19:58.669428 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:58.669432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:58.669504 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:58.669597 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:58.670887 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.671428 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.672561 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.672777 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.672890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:58.675816 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:19:58.675819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:19:58.675822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:19:58.675848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:19:58.677487 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:58.678071 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:58.678075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:58.678077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:58.678161 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:19:58.678518 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:19:58.678521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:19:58.678523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:19:58.678710 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:19:58.678818 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 08:19:58.679353 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 08:19:58.679477 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 08:19:58.680471 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:19:58.667591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:41353 (local address 127.12.13.3:53021)
0808 08:19:58.668276 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0808 08:19:58.668347 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 08:19:58.669182 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:19:58.669786 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:19:58.669793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:19:58.670232 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:19:58.670724 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:58.670738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.671553 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.671556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:58.672431 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:19:58.672440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:19:58.672934 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:19:58.672938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:19:58.673112 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:19:58.673771 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:19:58.673787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:19:58.675633 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:19:58.677607 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:58.677613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:58.677629 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:58.677963 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:58.678258 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:19:58.678261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:19:58.678263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:19:58.678412 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:19:58.678843 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:19:58.678850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:19:58.679094 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 08:19:58.679281 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 08:19:58.679612 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":123,"threads_started":1}
I20260808 08:19:58.681636 16592 heartbeater.cc:347] Connected to a master server at 127.12.13.62:41353
I20260808 08:19:58.681977 16592 heartbeater.cc:464] Registering TS with master...
I20260808 08:19:58.682577 16592 heartbeater.cc:511] Master 127.12.13.62:41353 requested a full tablet report, sending...
I20260808 08:19:58.683650 15892 ts_manager.cc:194] Registered new tserver with Master: 0d648fe08a1c4c718adf25814c0fc2c3 (127.12.13.3:40207)
I20260808 08:19:58.684301 15892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:53021
I20260808 08:19:58.688549 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 08:19:58.702996 15955 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:19:58.701563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.12.13.62:41353)
0808 08:19:58.701699 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:19:58.701703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:19:58.701978 (+ 275us) server_negotiation.cc:407] Connection header received
0808 08:19:58.702107 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:19:58.702109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:19:58.702153 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:19:58.702228 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:19:58.702886 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37866: BlockingRecv error: recv got EOF from 127.0.0.1:37866 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:19:58.704087 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 15992
I20260808 08:19:58.717952 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 16192
I20260808 08:19:58.729513 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 16402
I20260808 08:19:58.736887 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 15845
2026-08-08T08:19:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2509 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[16609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[16609](info): set up 2 sockets
Aug 08 08:19:58 dist-test-slave-6ntk krb5kdc[16609](info): commencing operation
krb5kdc: starting...
W20260808 08:19:59.997093 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:00 dist-test-slave-6ntk krb5kdc[16609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:00.137557 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:38163
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:38163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:00.263494 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:00.263798 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:00.263839 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:00.267980 16629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:00.268054 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:00.268074 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:00.268091 16629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:00.268163 16629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:00.274003 16629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:38163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:38163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:00.275204 16629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:00.276355 16629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:00.285346 16637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:00.285312 16648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:00.285334 16638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:00.285789 16629 server_base.cc:1034] running on GCE node
I20260808 08:20:00.286465 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:00.287593 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:00.288823 16629 hybrid_clock.cc:648] HybridClock initialized: now 1786177200288805 us; error 29 us; skew 500 ppm
Aug 08 08:20:00 dist-test-slave-6ntk krb5kdc[16609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177200, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:00.291810 16629 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:00.292721 16629 webserver.cc:460] Webserver started at http://127.12.13.62:35139/ using document root <none> and password file <none>
I20260808 08:20:00.293311 16629 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:00.293407 16629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:00.293653 16629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:00.295806 16629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "9b75ad68f7b54781a9bd28f8010f1d18"
format_stamp: "Formatted at 2026-08-08 08:20:00 on dist-test-slave-6ntk"
server_key: "14d91f13b754991cf8465ef7d6a5d9f4"
server_key_iv: "942a35b9236f58fe671b9df9489546d6"
server_key_version: "encryptionkey@0"
I20260808 08:20:00.296404 16629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "9b75ad68f7b54781a9bd28f8010f1d18"
format_stamp: "Formatted at 2026-08-08 08:20:00 on dist-test-slave-6ntk"
server_key: "14d91f13b754991cf8465ef7d6a5d9f4"
server_key_iv: "942a35b9236f58fe671b9df9489546d6"
server_key_version: "encryptionkey@0"
I20260808 08:20:00.300304 16629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:20:00.302783 16663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:00.304054 16629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:00.304214 16629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "9b75ad68f7b54781a9bd28f8010f1d18"
format_stamp: "Formatted at 2026-08-08 08:20:00 on dist-test-slave-6ntk"
server_key: "14d91f13b754991cf8465ef7d6a5d9f4"
server_key_iv: "942a35b9236f58fe671b9df9489546d6"
server_key_version: "encryptionkey@0"
I20260808 08:20:00.304327 16629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:00.316587 16629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:00.319768 16629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:00.320001 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:00.328308 16629 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:38163
I20260808 08:20:00.328619 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:38163 every 8 connection(s)
I20260808 08:20:00.329409 16629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:00.332460 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:00.335500 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 16629
I20260808 08:20:00.335634 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:00.335940 12340 external_mini_cluster.cc:1428] Setting key 3ef335399d7eb336d26c74ddfc8ff3de
I20260808 08:20:00.339053 16752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18: Bootstrap starting.
I20260808 08:20:00.341885 16752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:00.342765 16752 log.cc:824] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:00.345341 16752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18: No bootstrap required, opened a new log
Aug 08 08:20:00 dist-test-slave-6ntk krb5kdc[16609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:00.350575 16752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38163 } }
I20260808 08:20:00.350884 16752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:00.350947 16752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b75ad68f7b54781a9bd28f8010f1d18, State: Initialized, Role: FOLLOWER
I20260808 08:20:00.351464 16752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38163 } }
I20260808 08:20:00.351640 16752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:00.351716 16752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:00.351723 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:00.337310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.12.13.62:38163)
0808 08:20:00.337746 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:00.337763 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:00.337803 (+ 40us) server_negotiation.cc:407] Connection header received
0808 08:20:00.338553 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:00.338573 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:00.338921 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:00.339287 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:00.340220 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:00.341223 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:00.342477 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:00.343025 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:00.343148 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:00.346209 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:00.346216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:00.346233 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:00.346265 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:00.348420 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:00.348887 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:00.348893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:00.348900 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:00.349051 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:00.349324 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:00.349328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:00.349330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:00.349935 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:00.350158 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:00.350355 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 08:20:00.350673 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":149,"threads_started":1}
I20260808 08:20:00.351847 16752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:00.352880 16752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38163 } }
I20260808 08:20:00.353257 16752 leader_election.cc:302] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b75ad68f7b54781a9bd28f8010f1d18; no voters:
I20260808 08:20:00.353627 16752 leader_election.cc:288] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:00.353752 16758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:00.354019 16758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [term 1 LEADER]: Becoming Leader. State: Replica: 9b75ad68f7b54781a9bd28f8010f1d18, State: Running, Role: LEADER
I20260808 08:20:00.354382 16758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38163 } }
I20260808 08:20:00.355132 16752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:00.355321 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:20:00.356000 16760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38163 } } }
I20260808 08:20:00.356151 16760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:00.356431 16760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b75ad68f7b54781a9bd28f8010f1d18. Latest consensus state: current_term: 1 leader_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75ad68f7b54781a9bd28f8010f1d18" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38163 } } }
I20260808 08:20:00.356531 16760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:00.357018 16771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:00.360468 16771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:00.366338 16771 catalog_manager.cc:1347] Generated new cluster ID: 675bdea9a5574a0999513ac916032e57
I20260808 08:20:00.366429 16771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:00.376065 16771 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:00.377343 16771 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:00.385181 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:00.380086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.12.13.62:38163)
0808 08:20:00.380213 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:00.380218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:00.380492 (+ 274us) server_negotiation.cc:407] Connection header received
0808 08:20:00.380642 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:00.380645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:00.380699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:00.380789 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:00.381332 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:00.381950 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:00.382869 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:00.383118 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:00.383231 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:00.384179 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:00.384183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:00.384186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:00.384670 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:00.384700 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:00.384718 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:00.384823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:00.384949 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 08:20:00.385017 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:20:00.389705 16771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b75ad68f7b54781a9bd28f8010f1d18: Generated new TSK 0
I20260808 08:20:00.390460 16771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:00.399423 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 16629
2026-08-08T08:20:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1657 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:00 dist-test-slave-6ntk krb5kdc[16793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:00 dist-test-slave-6ntk krb5kdc[16793](info): set up 2 sockets
Aug 08 08:20:00 dist-test-slave-6ntk krb5kdc[16793](info): commencing operation
krb5kdc: starting...
W20260808 08:20:01.663172 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:01 dist-test-slave-6ntk krb5kdc[16793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:01.806456 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36825
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:01.925956 16821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:01.926297 16821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:01.926404 16821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:01.930585 16821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:01.930681 16821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:01.930718 16821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:01.930775 16821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:01.930835 16821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:01.936499 16821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:01.937727 16821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:01.938901 16821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:01.947980 16832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:01.948156 16821 server_base.cc:1034] running on GCE node
W20260808 08:20:01.947990 16829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:01.948064 16828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:01.949010 16821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:01.950278 16821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:01.951453 16821 hybrid_clock.cc:648] HybridClock initialized: now 1786177201951440 us; error 30 us; skew 500 ppm
Aug 08 08:20:01 dist-test-slave-6ntk krb5kdc[16793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177201, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:01.959398 16821 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:01.960466 16821 webserver.cc:460] Webserver started at http://127.12.13.62:38337/ using document root <none> and password file <none>
I20260808 08:20:01.961040 16821 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:01.961102 16821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:01.961280 16821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:01.963500 16821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "8af4eb7c8e6348e4abe3dde897c5ef83"
format_stamp: "Formatted at 2026-08-08 08:20:01 on dist-test-slave-6ntk"
server_key: "7ded495a7042be38bbb8069e0b4bde1d"
server_key_iv: "600061a53161ddbffd59a60bfccc1548"
server_key_version: "encryptionkey@0"
I20260808 08:20:01.964051 16821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "8af4eb7c8e6348e4abe3dde897c5ef83"
format_stamp: "Formatted at 2026-08-08 08:20:01 on dist-test-slave-6ntk"
server_key: "7ded495a7042be38bbb8069e0b4bde1d"
server_key_iv: "600061a53161ddbffd59a60bfccc1548"
server_key_version: "encryptionkey@0"
I20260808 08:20:01.968207 16821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:01.970813 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:01.972172 16821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:01.972342 16821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "8af4eb7c8e6348e4abe3dde897c5ef83"
format_stamp: "Formatted at 2026-08-08 08:20:01 on dist-test-slave-6ntk"
server_key: "7ded495a7042be38bbb8069e0b4bde1d"
server_key_iv: "600061a53161ddbffd59a60bfccc1548"
server_key_version: "encryptionkey@0"
I20260808 08:20:01.972470 16821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:01.996541 16821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:01.999967 16821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:02.000262 16821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:02.008463 16821 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36825
I20260808 08:20:02.008862 16918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36825 every 8 connection(s)
I20260808 08:20:02.009568 16821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:02.012761 16920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:02.014312 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 16821
I20260808 08:20:02.014456 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:02.014832 12340 external_mini_cluster.cc:1428] Setting key 57c763705a68941291922cb42161f437
I20260808 08:20:02.019815 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83: Bootstrap starting.
I20260808 08:20:02.022250 16920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:02.023149 16920 log.cc:824] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:02.025707 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83: No bootstrap required, opened a new log
I20260808 08:20:02.030664 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.016440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.12.13.62:36825)
0808 08:20:02.017057 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:02.017074 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:02.017112 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:20:02.017756 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:02.017773 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:02.018207 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:02.018560 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:02.019433 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.020484 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.021377 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.021927 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.022098 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.025188 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:02.025193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:02.025207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:02.025235 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:02.027166 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.027710 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.027715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.027720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.027849 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.028214 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.028218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.028219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.028914 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:02.029119 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:02.029434 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 08:20:02.029780 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":277,"threads_started":1}
I20260808 08:20:02.030594 16920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36825 } }
I20260808 08:20:02.030851 16920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:02.030915 16920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af4eb7c8e6348e4abe3dde897c5ef83, State: Initialized, Role: FOLLOWER
I20260808 08:20:02.031365 16920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36825 } }
I20260808 08:20:02.031492 16920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:02.031582 16920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:02.031703 16920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:02.033023 16920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36825 } }
I20260808 08:20:02.033448 16920 leader_election.cc:302] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af4eb7c8e6348e4abe3dde897c5ef83; no voters:
I20260808 08:20:02.033821 16920 leader_election.cc:288] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:02.033977 16926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:02.034246 16926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [term 1 LEADER]: Becoming Leader. State: Replica: 8af4eb7c8e6348e4abe3dde897c5ef83, State: Running, Role: LEADER
I20260808 08:20:02.034649 16926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36825 } }
I20260808 08:20:02.035181 16920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:02.036449 16928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af4eb7c8e6348e4abe3dde897c5ef83. Latest consensus state: current_term: 1 leader_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36825 } } }
I20260808 08:20:02.036449 16927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af4eb7c8e6348e4abe3dde897c5ef83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36825 } } }
I20260808 08:20:02.036610 16928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:02.036635 16927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:02.037349 16939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:02.040686 16939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:02.046371 16939 catalog_manager.cc:1347] Generated new cluster ID: 6f97258fe8c44b70ada842f2717a18c6
I20260808 08:20:02.046451 16939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:02.071738 16939 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:02.073510 16939 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:02.085012 16939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8af4eb7c8e6348e4abe3dde897c5ef83: Generated new TSK 0
I20260808 08:20:02.085837 16939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:20:02.121740 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36825
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:02.246268 16957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:02.246558 16957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:02.246603 16957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:02.250691 16957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:02.250748 16957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:02.250806 16957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:02.256067 16957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:02.257143 16957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:02.258199 16957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:02.268440 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:02.268440 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:02.268497 16969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:02.268839 16957 server_base.cc:1034] running on GCE node
I20260808 08:20:02.269325 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:02.270121 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:02.271291 16957 hybrid_clock.cc:648] HybridClock initialized: now 1786177202271276 us; error 25 us; skew 500 ppm
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:02.274179 16957 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:20:02.275087 16957 webserver.cc:460] Webserver started at http://127.12.13.1:45757/ using document root <none> and password file <none>
I20260808 08:20:02.275739 16957 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:02.275857 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:02.276093 16957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:02.278326 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "279edb2150ad48f791e5e91e4a7331d4"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "d340a8fc464e12e9668d635c08817b9b"
server_key_iv: "ea4980217f2df8d92d5ddf8df3ccb671"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.278910 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "279edb2150ad48f791e5e91e4a7331d4"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "d340a8fc464e12e9668d635c08817b9b"
server_key_iv: "ea4980217f2df8d92d5ddf8df3ccb671"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.283059 16957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:02.285615 16983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.286978 16957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 08:20:02.287125 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "279edb2150ad48f791e5e91e4a7331d4"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "d340a8fc464e12e9668d635c08817b9b"
server_key_iv: "ea4980217f2df8d92d5ddf8df3ccb671"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.287243 16957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:02.299675 16957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:02.302811 16957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:02.303040 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:02.303746 16957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:02.304649 16957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:02.304721 16957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.304796 16957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:02.304844 16957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.314236 16957 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:34497
I20260808 08:20:02.314255 17168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:34497 every 8 connection(s)
I20260808 08:20:02.315214 16957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:02.320088 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 16957
I20260808 08:20:02.320195 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:02.320506 12340 external_mini_cluster.cc:1428] Setting key f96a82d66c6438c34ca7497622ab51b1
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:02.331686 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.317198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59797 (local address 127.12.13.62:36825)
0808 08:20:02.317356 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:02.317361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:02.318251 (+ 890us) server_negotiation.cc:407] Connection header received
0808 08:20:02.319670 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:02.319673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:02.319737 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:02.319820 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:02.321296 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.322248 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.324052 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.324278 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.324374 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.327209 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:02.327216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:02.327222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:02.327277 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:02.329205 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.329813 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.329820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.329823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.329952 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.330345 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.330348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.330351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.330616 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:02.330727 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:02.331307 (+ 580us) server_negotiation.cc:299] Negotiation successful
0808 08:20:02.331452 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:02.332597 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.317563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36825 (local address 127.12.13.1:59797)
0808 08:20:02.318093 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:02.318138 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:02.319448 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:02.320001 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:02.320008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:02.320496 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:02.321112 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:02.321131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.322426 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.322431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:02.323900 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:02.323917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.324449 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.324456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:02.324695 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.325375 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:02.325392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:02.327009 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:02.329370 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:02.329376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:02.329391 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:02.329694 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:02.330068 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:02.330071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:02.330072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:02.330235 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:02.330737 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:02.330743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:02.331039 (+ 296us) client_negotiation.cc:769] Sending connection context
0808 08:20:02.331214 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 08:20:02.331510 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260808 08:20:02.334201 17169 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36825
I20260808 08:20:02.334476 17169 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:02.335039 17169 heartbeater.cc:511] Master 127.12.13.62:36825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:20:02.336957 16873 ts_manager.cc:194] Registered new tserver with Master: 279edb2150ad48f791e5e91e4a7331d4 (127.12.13.1:34497)
I20260808 08:20:02.338631 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:59797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:20:02.353479 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36825
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:02.472074 17179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:02.472384 17179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:02.472460 17179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:02.476567 17179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:02.476655 17179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:02.476742 17179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:02.482259 17179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:02.483421 17179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:02.484498 17179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:02.494858 17187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:02.494951 17179 server_base.cc:1034] running on GCE node
W20260808 08:20:02.494867 17190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:02.495287 17188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:02.495781 17179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:02.496480 17179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:02.497671 17179 hybrid_clock.cc:648] HybridClock initialized: now 1786177202497655 us; error 27 us; skew 500 ppm
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:02.500653 17179 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:20:02.501571 17179 webserver.cc:460] Webserver started at http://127.12.13.2:34981/ using document root <none> and password file <none>
I20260808 08:20:02.502249 17179 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:02.502344 17179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:02.502589 17179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:02.504671 17179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "a6efc7e078844a91ada9d31853fb2e2a"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "40a4eb9db2926673096551ecfd436921"
server_key_iv: "006781e201005ef369cb23e0f231cdf1"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.505275 17179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "a6efc7e078844a91ada9d31853fb2e2a"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "40a4eb9db2926673096551ecfd436921"
server_key_iv: "006781e201005ef369cb23e0f231cdf1"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.509277 17179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 08:20:02.511958 17198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.513228 17179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:02.513373 17179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "a6efc7e078844a91ada9d31853fb2e2a"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "40a4eb9db2926673096551ecfd436921"
server_key_iv: "006781e201005ef369cb23e0f231cdf1"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.513495 17179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:02.529131 17179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:02.532923 17179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:02.533154 17179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:02.533841 17179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:02.534816 17179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:02.534914 17179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.535012 17179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:02.535084 17179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.545097 17179 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:36327
I20260808 08:20:02.545120 17391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:36327 every 8 connection(s)
I20260808 08:20:02.546238 17179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:02.551047 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 17179
I20260808 08:20:02.551170 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:02.551497 12340 external_mini_cluster.cc:1428] Setting key 6a8ec1b798b84c59234f7bc6d769430b
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:02.561604 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.548202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:35209 (local address 127.12.13.62:36825)
0808 08:20:02.548424 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:02.548429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:02.549279 (+ 850us) server_negotiation.cc:407] Connection header received
0808 08:20:02.550865 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:02.550873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:02.550924 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:02.551000 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:02.552597 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.553257 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.555043 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.555289 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.555432 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.558032 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:02.558035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:02.558038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:02.558059 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:02.559507 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.560046 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.560049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.560050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.560125 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.560483 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.560486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.560487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.560675 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:02.560783 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:02.561288 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 08:20:02.561424 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:20:02.562577 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.548619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36825 (local address 127.12.13.2:35209)
0808 08:20:02.549107 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:02.549155 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:02.550588 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:02.551237 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:02.551246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:02.551785 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:02.552451 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:02.552467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.553417 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.553421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:02.554915 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:02.554931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.555482 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.555486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:02.555654 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.556321 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:02.556339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:02.557801 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:02.559627 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:02.559633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:02.559649 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:02.559924 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:02.560219 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:02.560223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:02.560225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:02.560379 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:02.560782 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:02.560789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:02.561045 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 08:20:02.561231 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 08:20:02.561559 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
I20260808 08:20:02.563776 17394 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36825
I20260808 08:20:02.564049 17394 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:02.564597 17394 heartbeater.cc:511] Master 127.12.13.62:36825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:20:02.565764 16873 ts_manager.cc:194] Registered new tserver with Master: a6efc7e078844a91ada9d31853fb2e2a (127.12.13.2:36327)
I20260808 08:20:02.566411 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:35209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:20:02.583451 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36825
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:02.708050 17400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:02.708341 17400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:02.708390 17400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:02.712457 17400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:02.712524 17400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:02.712581 17400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:02.718029 17400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:02.719089 17400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:02.720109 17400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:20:02.730508 17400 server_base.cc:1034] running on GCE node
W20260808 08:20:02.730460 17409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:02.730453 17411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:02.730473 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:02.731256 17400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:02.731973 17400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:02.733170 17400 hybrid_clock.cc:648] HybridClock initialized: now 1786177202733147 us; error 37 us; skew 500 ppm
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:02.736519 17400 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:20:02.737622 17400 webserver.cc:460] Webserver started at http://127.12.13.3:36895/ using document root <none> and password file <none>
I20260808 08:20:02.738301 17400 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:02.738377 17400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:02.738626 17400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:02.740839 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "1e7ed4f3e9ba44a78ca11540be82507b"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "68eb4115a26ac5a5044674c17234c0fd"
server_key_iv: "1dbb093cf7a34a0e1dfa819e51f5ab8c"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.741461 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "1e7ed4f3e9ba44a78ca11540be82507b"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "68eb4115a26ac5a5044674c17234c0fd"
server_key_iv: "1dbb093cf7a34a0e1dfa819e51f5ab8c"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.745407 17400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:20:02.748040 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.749377 17400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 08:20:02.749521 17400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "1e7ed4f3e9ba44a78ca11540be82507b"
format_stamp: "Formatted at 2026-08-08 08:20:02 on dist-test-slave-6ntk"
server_key: "68eb4115a26ac5a5044674c17234c0fd"
server_key_iv: "1dbb093cf7a34a0e1dfa819e51f5ab8c"
server_key_version: "encryptionkey@0"
I20260808 08:20:02.749636 17400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:02.770244 17400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:02.774005 17400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:02.774226 17400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:02.774829 17400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:02.775691 17400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:02.775748 17400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.775791 17400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:02.775812 17400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:02.785794 17400 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:39455
I20260808 08:20:02.785822 17605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:39455 every 8 connection(s)
I20260808 08:20:02.786867 17400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:02.792284 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 17400
I20260808 08:20:02.792420 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:02.792713 12340 external_mini_cluster.cc:1428] Setting key 42c16b3f8840ef8f2e6c5eeb581eead7
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:02.801083 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.788804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:58099 (local address 127.12.13.62:36825)
0808 08:20:02.788968 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:02.788972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:02.789846 (+ 874us) server_negotiation.cc:407] Connection header received
0808 08:20:02.791398 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:02.791401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:02.791451 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:02.791530 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:02.792944 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.793536 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.794844 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.795130 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.795219 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.797742 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:02.797745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:02.797747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:02.797773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:02.799212 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.799678 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.799681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.799682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.799756 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.800056 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.800058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.800059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.800235 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:02.800312 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:02.800821 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 08:20:02.800944 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 08:20:02.801903 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.789211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36825 (local address 127.12.13.3:58099)
0808 08:20:02.789682 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:02.789727 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:02.791170 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:02.791699 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:02.791706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:02.792171 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:02.792760 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:02.792775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.793673 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.793677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:02.794719 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:02.794734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.795267 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.795269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:02.795415 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.795979 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:02.796032 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:02.797593 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:02.799315 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:02.799320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:02.799332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:02.799582 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:02.799834 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:02.799837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:02.799838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:02.799976 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:02.800323 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:02.800328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:02.800554 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 08:20:02.800720 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 08:20:02.800980 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260808 08:20:02.803094 17606 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36825
I20260808 08:20:02.803333 17606 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:02.803824 17606 heartbeater.cc:511] Master 127.12.13.62:36825 requested a full tablet report, sending...
I20260808 08:20:02.804877 16873 ts_manager.cc:194] Registered new tserver with Master: 1e7ed4f3e9ba44a78ca11540be82507b (127.12.13.3:39455)
I20260808 08:20:02.805465 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:58099
I20260808 08:20:02.806816 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[16793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:02.834753 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:02.825423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.12.13.62:36825)
0808 08:20:02.825555 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:02.825559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:02.825783 (+ 224us) server_negotiation.cc:407] Connection header received
0808 08:20:02.825976 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:02.825979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:02.826025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:02.826101 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:02.826940 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.827458 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.828360 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:02.828564 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:02.828644 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:02.831411 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:02.831415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:02.831417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:02.831443 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:02.832932 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.833382 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.833390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.833391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.833469 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:02.833752 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:02.833756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:02.833758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:02.834066 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:02.834209 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:02.834455 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 08:20:02.834579 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 08:20:02.835206 16873 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45308
I20260808 08:20:02.836360 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 16957
I20260808 08:20:02.849339 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 17179
I20260808 08:20:02.862390 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 17400
I20260808 08:20:02.879803 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 16821
2026-08-08T08:20:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2480 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[17626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[17626](info): set up 2 sockets
Aug 08 08:20:02 dist-test-slave-6ntk krb5kdc[17626](info): commencing operation
krb5kdc: starting...
W20260808 08:20:04.184113 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:04 dist-test-slave-6ntk krb5kdc[17626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:04.333356 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35355
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:35355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:04.455456 17644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:04.455892 17644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:04.455986 17644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:04.460161 17644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:04.460317 17644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:04.460356 17644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:04.460394 17644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:04.460429 17644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:04.466146 17644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:35355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:04.467299 17644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:04.468353 17644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:04.477833 17657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:04.477890 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:04.477916 17654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:04.478076 17644 server_base.cc:1034] running on GCE node
I20260808 08:20:04.478834 17644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:04.480105 17644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:04.481307 17644 hybrid_clock.cc:648] HybridClock initialized: now 1786177204481293 us; error 26 us; skew 500 ppm
Aug 08 08:20:04 dist-test-slave-6ntk krb5kdc[17626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177204, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:04.488139 17644 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:04.489243 17644 webserver.cc:460] Webserver started at http://127.12.13.62:46221/ using document root <none> and password file <none>
I20260808 08:20:04.489997 17644 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:04.490103 17644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:04.490373 17644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:04.492799 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "342db86958cc4a6cbc143177c3019cc1"
format_stamp: "Formatted at 2026-08-08 08:20:04 on dist-test-slave-6ntk"
server_key: "975ed74e2ed3c96f1e5e18ebb0db8c0e"
server_key_iv: "b65c2726557ed8cb652052e53e535f6e"
server_key_version: "encryptionkey@0"
I20260808 08:20:04.493446 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "342db86958cc4a6cbc143177c3019cc1"
format_stamp: "Formatted at 2026-08-08 08:20:04 on dist-test-slave-6ntk"
server_key: "975ed74e2ed3c96f1e5e18ebb0db8c0e"
server_key_iv: "b65c2726557ed8cb652052e53e535f6e"
server_key_version: "encryptionkey@0"
I20260808 08:20:04.497678 17644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 08:20:04.501204 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:04.503295 17644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260808 08:20:04.503458 17644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "342db86958cc4a6cbc143177c3019cc1"
format_stamp: "Formatted at 2026-08-08 08:20:04 on dist-test-slave-6ntk"
server_key: "975ed74e2ed3c96f1e5e18ebb0db8c0e"
server_key_iv: "b65c2726557ed8cb652052e53e535f6e"
server_key_version: "encryptionkey@0"
I20260808 08:20:04.503624 17644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:04.552392 17644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:04.555790 17644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:04.556036 17644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:04.563967 17644 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:35355
I20260808 08:20:04.563997 17759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:35355 every 8 connection(s)
I20260808 08:20:04.565418 17644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:04.570063 17762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:04.573478 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 17644
I20260808 08:20:04.573618 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:04.574025 12340 external_mini_cluster.cc:1428] Setting key bd74fd6404f9e345347432c19af1a624
I20260808 08:20:04.580067 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1: Bootstrap starting.
I20260808 08:20:04.583429 17762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:04.584376 17762 log.cc:824] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:20:04 dist-test-slave-6ntk krb5kdc[17626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:04.588399 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1: No bootstrap required, opened a new log
I20260808 08:20:04.593036 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:04.575633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.12.13.62:35355)
0808 08:20:04.576178 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:04.576200 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:04.576249 (+ 49us) server_negotiation.cc:407] Connection header received
0808 08:20:04.577129 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:04.577155 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:04.577622 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:04.578106 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:04.579149 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:04.580630 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:04.582031 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:04.582750 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:04.582924 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:04.586786 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:04.586793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:04.586812 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:04.586845 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:04.589570 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:04.590135 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:04.590140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:04.590145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:04.590339 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:04.590762 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:04.590765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:04.590767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:04.591217 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:04.591493 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:04.591837 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 08:20:04.592165 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":151,"threads_started":1}
I20260808 08:20:04.594431 17762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342db86958cc4a6cbc143177c3019cc1" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35355 } }
I20260808 08:20:04.594779 17762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:04.594856 17762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342db86958cc4a6cbc143177c3019cc1, State: Initialized, Role: FOLLOWER
I20260808 08:20:04.595434 17762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342db86958cc4a6cbc143177c3019cc1" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35355 } }
I20260808 08:20:04.595618 17762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:04.595682 17762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:04.595780 17762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:04.597424 17762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342db86958cc4a6cbc143177c3019cc1" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35355 } }
I20260808 08:20:04.597975 17762 leader_election.cc:302] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342db86958cc4a6cbc143177c3019cc1; no voters:
I20260808 08:20:04.598650 17762 leader_election.cc:288] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:04.598798 17768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:04.600356 17768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 342db86958cc4a6cbc143177c3019cc1, State: Running, Role: LEADER
I20260808 08:20:04.600725 17768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342db86958cc4a6cbc143177c3019cc1" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35355 } }
I20260808 08:20:04.601548 17762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:04.602720 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342db86958cc4a6cbc143177c3019cc1. Latest consensus state: current_term: 1 leader_uuid: "342db86958cc4a6cbc143177c3019cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342db86958cc4a6cbc143177c3019cc1" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35355 } } }
I20260808 08:20:04.602753 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342db86958cc4a6cbc143177c3019cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342db86958cc4a6cbc143177c3019cc1" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35355 } } }
I20260808 08:20:04.603003 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:04.603087 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:04.603430 17784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:04.607169 17784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:04.614041 17784 catalog_manager.cc:1347] Generated new cluster ID: 2559ef1cc3434221a21064db2b46f847
I20260808 08:20:04.614126 17784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:04.624701 17784 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:04.626171 17784 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:04.643440 17784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342db86958cc4a6cbc143177c3019cc1: Generated new TSK 0
I20260808 08:20:04.644707 17784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:20:04.686165 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35355
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:04.807878 17802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:04.808192 17802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:04.808300 17802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:04.812711 17802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:04.812806 17802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:04.812893 17802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:04.818382 17802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:04.819561 17802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:04.820649 17802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:04.831108 17810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:04.831112 17812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:04.831327 17802 server_base.cc:1034] running on GCE node
W20260808 08:20:04.831111 17808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:04.831935 17802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:04.832666 17802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:04.833910 17802 hybrid_clock.cc:648] HybridClock initialized: now 1786177204833854 us; error 29 us; skew 500 ppm
Aug 08 08:20:04 dist-test-slave-6ntk krb5kdc[17626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177204, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:04.837007 17802 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:20:04.838165 17802 webserver.cc:460] Webserver started at http://127.12.13.1:37423/ using document root <none> and password file <none>
I20260808 08:20:04.838764 17802 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:04.838828 17802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:04.839010 17802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:04.841202 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "79e18641c6f54c12a660de1dd7d8459f"
format_stamp: "Formatted at 2026-08-08 08:20:04 on dist-test-slave-6ntk"
server_key: "70274be87de9045258cdd6f0f5bf1306"
server_key_iv: "ff3115c97d36f5152e494ab6cd96e224"
server_key_version: "encryptionkey@0"
I20260808 08:20:04.841769 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "79e18641c6f54c12a660de1dd7d8459f"
format_stamp: "Formatted at 2026-08-08 08:20:04 on dist-test-slave-6ntk"
server_key: "70274be87de9045258cdd6f0f5bf1306"
server_key_iv: "ff3115c97d36f5152e494ab6cd96e224"
server_key_version: "encryptionkey@0"
I20260808 08:20:04.846067 17802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:04.848635 17823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:04.850067 17802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:04.850209 17802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "79e18641c6f54c12a660de1dd7d8459f"
format_stamp: "Formatted at 2026-08-08 08:20:04 on dist-test-slave-6ntk"
server_key: "70274be87de9045258cdd6f0f5bf1306"
server_key_iv: "ff3115c97d36f5152e494ab6cd96e224"
server_key_version: "encryptionkey@0"
I20260808 08:20:04.850337 17802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:04.872437 17802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:04.876101 17802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:04.876322 17802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:04.876981 17802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:04.877925 17802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:04.877997 17802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:04.878074 17802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:04.878108 17802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:04.887735 17802 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:37855
I20260808 08:20:04.887760 18004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:37855 every 8 connection(s)
I20260808 08:20:04.888700 17802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:04.894647 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 17802
I20260808 08:20:04.894806 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:04.895112 12340 external_mini_cluster.cc:1428] Setting key 5a0d61c257c32e7872e7fcdadf95392c
Aug 08 08:20:04 dist-test-slave-6ntk krb5kdc[17626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177204, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:04.903930 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:04.890679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:39279 (local address 127.12.13.62:35355)
0808 08:20:04.890840 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:04.890851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:04.891773 (+ 922us) server_negotiation.cc:407] Connection header received
0808 08:20:04.893217 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:04.893220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:04.893270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:04.893351 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:04.894663 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:04.895375 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:04.896573 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:04.896812 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:04.896932 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:04.899670 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:04.899674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:04.899675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:04.899700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:04.901240 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:04.902060 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:04.902063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:04.902064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:04.902147 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:04.902665 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:04.902673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:04.902674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:04.902939 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:04.903058 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:04.903612 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 08:20:04.903751 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:20:04.904887 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:04.891122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35355 (local address 127.12.13.1:39279)
0808 08:20:04.891600 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:04.891647 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:04.892995 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:04.893522 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:04.893529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:04.893967 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:04.894487 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:04.894502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:04.895518 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:04.895527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:04.896431 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:04.896440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:04.896982 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:04.896986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:04.897168 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:04.897824 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:04.897848 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:04.899485 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:04.901459 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:04.901469 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:04.901492 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:04.901951 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:04.902249 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:04.902254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:04.902257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:04.902510 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:04.903058 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:04.903066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:04.903350 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 08:20:04.903556 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 08:20:04.903992 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:04.906544 18006 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35355
I20260808 08:20:04.906862 18006 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:04.907514 18006 heartbeater.cc:511] Master 127.12.13.62:35355 requested a full tablet report, sending...
I20260808 08:20:04.909533 17704 ts_manager.cc:194] Registered new tserver with Master: 79e18641c6f54c12a660de1dd7d8459f (127.12.13.1:37855)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:20:04.911178 17704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:39279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:20:04.930414 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35355
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:05.052516 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:05.052817 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:05.052863 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:05.058102 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:05.058193 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:05.058314 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:05.064678 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:05.065949 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:05.067103 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:05.078224 18028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:05.078249 18025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:05.078254 18024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:05.078619 18017 server_base.cc:1034] running on GCE node
I20260808 08:20:05.079113 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:05.079870 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:05.081053 18017 hybrid_clock.cc:648] HybridClock initialized: now 1786177205081016 us; error 50 us; skew 500 ppm
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:05.084215 18017 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:20:05.085512 18017 webserver.cc:460] Webserver started at http://127.12.13.2:41405/ using document root <none> and password file <none>
I20260808 08:20:05.086376 18017 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:05.086452 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:05.086710 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:05.089215 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "b71a4ee4182a432bb2a14f8e8b8c2f09"
format_stamp: "Formatted at 2026-08-08 08:20:05 on dist-test-slave-6ntk"
server_key: "6e766928d23501365fff9fdaa619185e"
server_key_iv: "74eebb6d0b6bb79a15a1618093db6a99"
server_key_version: "encryptionkey@0"
I20260808 08:20:05.089934 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "b71a4ee4182a432bb2a14f8e8b8c2f09"
format_stamp: "Formatted at 2026-08-08 08:20:05 on dist-test-slave-6ntk"
server_key: "6e766928d23501365fff9fdaa619185e"
server_key_iv: "74eebb6d0b6bb79a15a1618093db6a99"
server_key_version: "encryptionkey@0"
I20260808 08:20:05.094425 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:05.097177 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:05.098729 18017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:05.098893 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "b71a4ee4182a432bb2a14f8e8b8c2f09"
format_stamp: "Formatted at 2026-08-08 08:20:05 on dist-test-slave-6ntk"
server_key: "6e766928d23501365fff9fdaa619185e"
server_key_iv: "74eebb6d0b6bb79a15a1618093db6a99"
server_key_version: "encryptionkey@0"
I20260808 08:20:05.099032 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:05.118079 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:05.121748 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:05.122059 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:05.122761 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:05.123718 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:05.123798 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:05.123878 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:05.123929 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:05.133513 18017 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:41815
I20260808 08:20:05.133536 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:41815 every 8 connection(s)
I20260808 08:20:05.134663 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:05.139534 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 18017
I20260808 08:20:05.139724 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:05.140015 12340 external_mini_cluster.cc:1428] Setting key 445c4302f81f2b1c75d5b5f08c333274
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:05.151678 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:05.136729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:56429 (local address 127.12.13.62:35355)
0808 08:20:05.136896 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:05.136900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:05.137921 (+ 1021us) server_negotiation.cc:407] Connection header received
0808 08:20:05.139773 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:05.139779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:05.139859 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:05.139987 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:05.141580 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.142516 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.143922 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.144342 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.144440 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:05.147576 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:05.147580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:05.147582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:05.147608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:05.149217 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.149805 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.149809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.149810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.149979 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.150379 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.150382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.150384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.150620 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:05.150736 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:05.151343 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 08:20:05.151496 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:20:05.152690 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:05.137135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35355 (local address 127.12.13.2:56429)
0808 08:20:05.137690 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:05.137743 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:05.139441 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:05.140204 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:05.140212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:05.140782 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:05.141356 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:05.141372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.142713 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.142716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:05.143780 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:05.143791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.144534 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.144540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:05.144784 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:05.145561 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:05.145586 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:05.147363 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:05.149368 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:05.149375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:05.149394 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:05.149687 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:05.150082 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:05.150085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:05.150087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:05.150251 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:05.150752 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:05.150761 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:05.151059 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 08:20:05.151295 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 08:20:05.151686 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":127,"threads_started":1}
I20260808 08:20:05.154280 18233 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35355
I20260808 08:20:05.154613 18233 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:20:05.155447 18233 heartbeater.cc:511] Master 127.12.13.62:35355 requested a full tablet report, sending...
I20260808 08:20:05.156775 17704 ts_manager.cc:194] Registered new tserver with Master: b71a4ee4182a432bb2a14f8e8b8c2f09 (127.12.13.2:41815)
I20260808 08:20:05.157442 17704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:56429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:20:05.177192 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35355
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:05.305341 18239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:05.305636 18239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:05.305719 18239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:05.310348 18239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:05.310421 18239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:05.310508 18239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:05.315927 18239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:05.317060 18239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:05.318224 18239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:05.328562 18255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:05.328516 18252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:05.328552 18257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:05.328532 18239 server_base.cc:1034] running on GCE node
I20260808 08:20:05.329316 18239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:05.330081 18239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:05.331256 18239 hybrid_clock.cc:648] HybridClock initialized: now 1786177205331215 us; error 53 us; skew 500 ppm
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:05.334299 18239 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:20:05.335263 18239 webserver.cc:460] Webserver started at http://127.12.13.3:43383/ using document root <none> and password file <none>
I20260808 08:20:05.335813 18239 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:05.335887 18239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:05.336066 18239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:05.338258 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "eb0a6db2936a4f26bc5cf66b12e858ac"
format_stamp: "Formatted at 2026-08-08 08:20:05 on dist-test-slave-6ntk"
server_key: "5468bd5c73fdf97f69ede0d802a19e7c"
server_key_iv: "73e82200aa8918890140e3032fbd9dfa"
server_key_version: "encryptionkey@0"
I20260808 08:20:05.338793 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "eb0a6db2936a4f26bc5cf66b12e858ac"
format_stamp: "Formatted at 2026-08-08 08:20:05 on dist-test-slave-6ntk"
server_key: "5468bd5c73fdf97f69ede0d802a19e7c"
server_key_iv: "73e82200aa8918890140e3032fbd9dfa"
server_key_version: "encryptionkey@0"
I20260808 08:20:05.342960 18239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 08:20:05.345335 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:05.346676 18239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:05.346773 18239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "eb0a6db2936a4f26bc5cf66b12e858ac"
format_stamp: "Formatted at 2026-08-08 08:20:05 on dist-test-slave-6ntk"
server_key: "5468bd5c73fdf97f69ede0d802a19e7c"
server_key_iv: "73e82200aa8918890140e3032fbd9dfa"
server_key_version: "encryptionkey@0"
I20260808 08:20:05.346868 18239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:05.361572 18239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:05.364770 18239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:05.364962 18239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:05.365552 18239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:05.366536 18239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:05.366587 18239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:05.366631 18239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:05.366657 18239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:05.376073 18239 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:41331
I20260808 08:20:05.376098 18453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:41331 every 8 connection(s)
I20260808 08:20:05.377188 18239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:05.386003 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 18239
I20260808 08:20:05.386155 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:05.386531 12340 external_mini_cluster.cc:1428] Setting key 7e42977659d7d35543c7caf2288bb456
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:05.391994 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:05.379173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:60473 (local address 127.12.13.62:35355)
0808 08:20:05.379369 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:05.379374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:05.380299 (+ 925us) server_negotiation.cc:407] Connection header received
0808 08:20:05.381709 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:05.381711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:05.381761 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:05.381840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:05.383191 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.383729 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.384929 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.385153 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.385236 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:05.388244 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:05.388248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:05.388251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:05.388284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:05.389819 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.390356 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.390361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.390366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.390459 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.390822 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.390827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.390829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.391043 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:05.391167 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:05.391680 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 08:20:05.391809 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:20:05.392822 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:05.379643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35355 (local address 127.12.13.3:60473)
0808 08:20:05.380127 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:05.380173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:05.381504 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:05.382125 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:05.382132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:05.382522 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:05.383037 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:05.383051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.383850 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.383853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:05.384814 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:05.384823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.385333 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.385336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:05.385475 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:05.386290 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:05.386313 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:05.388078 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:05.389960 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:05.389966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:05.389983 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:05.390248 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:05.390555 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:05.390559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:05.390561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:05.390721 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:05.391165 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:05.391172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:05.391436 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 08:20:05.391636 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 08:20:05.391972 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260808 08:20:05.394011 18455 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35355
I20260808 08:20:05.394279 18455 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:05.394860 18455 heartbeater.cc:511] Master 127.12.13.62:35355 requested a full tablet report, sending...
I20260808 08:20:05.396008 17704 ts_manager.cc:194] Registered new tserver with Master: eb0a6db2936a4f26bc5cf66b12e858ac (127.12.13.3:41331)
I20260808 08:20:05.396670 17704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:60473
I20260808 08:20:05.400899 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:20:05.431610 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:05.420711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.12.13.1:37855)
0808 08:20:05.421038 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:05.421048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:05.421084 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:20:05.421139 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:05.421144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:05.421346 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:05.421480 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:05.422405 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.423174 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.424143 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.424505 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.424586 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:05.427878 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:05.427893 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:05.427916 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:05.427956 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:05.429652 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.430198 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.430204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.430205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.430295 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.430569 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.430571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.430572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.430838 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:05.431007 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:05.431180 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 08:20:05.431382 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":119,"threads_started":1}
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[17626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:05.444824 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:05.435310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.12.13.62:35355)
0808 08:20:05.435503 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:05.435507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:05.435524 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:20:05.435689 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:05.435692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:05.435746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:05.435840 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:05.436605 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.437150 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.438188 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:05.438434 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:05.438534 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:05.441426 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:05.441429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:05.441431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:05.441463 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:05.442967 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.443432 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.443435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.443436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.443516 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:05.443808 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:05.443811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:05.443812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:05.444084 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:05.444207 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:05.444490 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 08:20:05.444603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260808 08:20:05.445394 17704 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598
I20260808 08:20:05.446151 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 17802
I20260808 08:20:05.463654 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 18017
I20260808 08:20:05.478507 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 18239
I20260808 08:20:05.493722 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 17644
2026-08-08T08:20:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2614 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[18473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[18473](info): set up 2 sockets
Aug 08 08:20:05 dist-test-slave-6ntk krb5kdc[18473](info): commencing operation
krb5kdc: starting...
W20260808 08:20:06.772289 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:06 dist-test-slave-6ntk krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:06.913574 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42877
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:07.034632 18492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:07.034956 18492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:07.035003 18492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:07.039202 18492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:07.039283 18492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:07.039304 18492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:07.039322 18492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:07.039345 18492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:07.045061 18492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:42877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:07.046360 18492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:07.047454 18492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:07.056365 18497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.056429 18503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.056437 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:07.056507 18492 server_base.cc:1034] running on GCE node
I20260808 08:20:07.057320 18492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:07.058565 18492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:07.059736 18492 hybrid_clock.cc:648] HybridClock initialized: now 1786177207059719 us; error 30 us; skew 500 ppm
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:07.067524 18492 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:07.068394 18492 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:07.068501 18492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:07.068768 18492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:07.071198 18492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "eac29ff2fbee4bdfb2674632ffea3fee"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "16ae25dc14f73cedafd0b13e69d66da7"
server_key_iv: "a3092b9c20df723f9e7f5c9e97e8eb7e"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.071825 18492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "eac29ff2fbee4bdfb2674632ffea3fee"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "16ae25dc14f73cedafd0b13e69d66da7"
server_key_iv: "a3092b9c20df723f9e7f5c9e97e8eb7e"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.076375 18492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:07.079442 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.080844 18492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:20:07.081023 18492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "eac29ff2fbee4bdfb2674632ffea3fee"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "16ae25dc14f73cedafd0b13e69d66da7"
server_key_iv: "a3092b9c20df723f9e7f5c9e97e8eb7e"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.081171 18492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:07.105137 18492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:07.108896 18492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:07.109136 18492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:07.116407 18492 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:42877
I20260808 08:20:07.116664 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:42877 every 8 connection(s)
I20260808 08:20:07.117138 18492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:07.120112 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:07.122309 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 18492
I20260808 08:20:07.122459 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:07.122766 12340 external_mini_cluster.cc:1428] Setting key 3c840ff63edd16c785fa9b1443fc478d
I20260808 08:20:07.127425 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee: Bootstrap starting.
I20260808 08:20:07.130182 18595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:07.131151 18595 log.cc:824] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:07.133611 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee: No bootstrap required, opened a new log
I20260808 08:20:07.138391 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.124269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.12.13.62:42877)
0808 08:20:07.124715 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.124732 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.124769 (+ 37us) server_negotiation.cc:407] Connection header received
0808 08:20:07.125430 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.125448 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.125781 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.126174 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:07.127055 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.128090 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.129274 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.129767 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.129974 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.132799 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:07.132805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:07.132823 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:07.132856 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:07.134999 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.135447 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.135453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.135460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.135612 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.135884 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.135887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.135889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.136537 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:07.136740 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.136953 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.137289 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":151,"threads_started":1}
I20260808 08:20:07.138708 18595 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42877 } }
I20260808 08:20:07.139020 18595 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:07.139075 18595 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac29ff2fbee4bdfb2674632ffea3fee, State: Initialized, Role: FOLLOWER
I20260808 08:20:07.139596 18595 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42877 } }
I20260808 08:20:07.139747 18595 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:07.139842 18595 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:07.139946 18595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:07.141127 18595 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42877 } }
I20260808 08:20:07.141501 18595 leader_election.cc:302] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac29ff2fbee4bdfb2674632ffea3fee; no voters:
I20260808 08:20:07.141901 18595 leader_election.cc:288] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:07.142102 18603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:07.142382 18603 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [term 1 LEADER]: Becoming Leader. State: Replica: eac29ff2fbee4bdfb2674632ffea3fee, State: Running, Role: LEADER
I20260808 08:20:07.142701 18603 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42877 } }
I20260808 08:20:07.143273 18595 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:07.144547 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42877 } } }
I20260808 08:20:07.144692 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:07.145000 18606 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac29ff2fbee4bdfb2674632ffea3fee. Latest consensus state: current_term: 1 leader_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac29ff2fbee4bdfb2674632ffea3fee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42877 } } }
I20260808 08:20:07.145104 18606 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:07.145223 18613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:07.148561 18613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:07.154637 18613 catalog_manager.cc:1347] Generated new cluster ID: 9f6e982860ff4e238b1a3321501c57a7
I20260808 08:20:07.154719 18613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:07.178901 18613 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:07.179983 18613 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:07.185664 18613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee: Generated new TSK 0
I20260808 08:20:07.186484 18613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:20:07.230221 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42877
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:07.351097 18629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:07.351400 18629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:07.351449 18629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:07.355525 18629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:07.355594 18629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:07.355656 18629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:07.361083 18629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:07.362358 18629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:07.363477 18629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:07.373503 18639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.373550 18636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.373562 18637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:07.373692 18629 server_base.cc:1034] running on GCE node
I20260808 08:20:07.374280 18629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:07.374981 18629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:07.376150 18629 hybrid_clock.cc:648] HybridClock initialized: now 1786177207376114 us; error 49 us; skew 500 ppm
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:07.379069 18629 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:20:07.379778 18629 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:07.379841 18629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:07.380012 18629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:07.382205 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "fb92f175625645df81f865323faadefc"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "ba9bbe9365a94e797da69ef1df6484e3"
server_key_iv: "821cceb5ea606cb005880475d55978ae"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.382782 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "fb92f175625645df81f865323faadefc"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "ba9bbe9365a94e797da69ef1df6484e3"
server_key_iv: "821cceb5ea606cb005880475d55978ae"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.386845 18629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 08:20:07.389693 18651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.390983 18629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260808 08:20:07.391084 18629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "fb92f175625645df81f865323faadefc"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "ba9bbe9365a94e797da69ef1df6484e3"
server_key_iv: "821cceb5ea606cb005880475d55978ae"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.391172 18629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:07.404953 18629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:07.408304 18629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:07.408474 18629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:07.409027 18629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:07.409862 18629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:07.409986 18629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.410030 18629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:07.410051 18629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.419339 18629 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:39949
I20260808 08:20:07.419359 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:39949 every 8 connection(s)
I20260808 08:20:07.419966 18629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:07.428181 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 18629
I20260808 08:20:07.428318 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:07.428663 12340 external_mini_cluster.cc:1428] Setting key 90b194b94f836453578cb4dbf54eaec9
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:07.435143 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.421844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:42421 (local address 127.12.13.62:42877)
0808 08:20:07.422019 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.422023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.422935 (+ 912us) server_negotiation.cc:407] Connection header received
0808 08:20:07.424311 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.424314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.424374 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.424454 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:07.425691 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.426346 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.427459 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.427680 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.427759 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.430578 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:07.430583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:07.430585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:07.430614 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:07.432307 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.433023 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.433025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.433027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.433105 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.433546 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.433551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.433555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.433823 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:07.433948 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.434796 (+ 848us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.434962 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 08:20:07.436189 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.422271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42877 (local address 127.12.13.1:42421)
0808 08:20:07.422743 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:07.422796 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:07.424048 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:07.424624 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:07.424631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:07.425021 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:07.425514 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.425527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.426478 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.426481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.427348 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.427358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.427813 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.427816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.427967 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.428646 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:07.428662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:07.430399 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:07.432458 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:07.432466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:07.432487 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:07.432913 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:07.433203 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:07.433207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:07.433210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:07.433413 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:07.434044 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:07.434052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:07.434435 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 08:20:07.434701 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 08:20:07.435072 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260808 08:20:07.437940 18822 heartbeater.cc:347] Connected to a master server at 127.12.13.62:42877
I20260808 08:20:07.438248 18822 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:07.438889 18822 heartbeater.cc:511] Master 127.12.13.62:42877 requested a full tablet report, sending...
I20260808 08:20:07.440861 18536 ts_manager.cc:194] Registered new tserver with Master: fb92f175625645df81f865323faadefc (127.12.13.1:39949)
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:07.442356 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:42421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:20:07.461611 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42877
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:07.582960 18831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:07.583271 18831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:07.583318 18831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:07.587675 18831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:07.587749 18831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:07.587862 18831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:07.593387 18831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:07.594635 18831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:07.595803 18831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:20:07.606477 18831 server_base.cc:1034] running on GCE node
W20260808 08:20:07.606359 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.606351 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.606364 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:07.607172 18831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:07.607954 18831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:07.609122 18831 hybrid_clock.cc:648] HybridClock initialized: now 1786177207609107 us; error 29 us; skew 500 ppm
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:07.612264 18831 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:20:07.613086 18831 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:07.613154 18831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:07.613341 18831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:07.615666 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "e5a853eb2797409591e364f4bbc6582d"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "431d86b54d74d21c9534538f9e85a2ce"
server_key_iv: "f89107d4b45497a96cde9b8b624022d4"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.616247 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "e5a853eb2797409591e364f4bbc6582d"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "431d86b54d74d21c9534538f9e85a2ce"
server_key_iv: "f89107d4b45497a96cde9b8b624022d4"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.620712 18831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 08:20:07.623908 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.625423 18831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:07.625600 18831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "e5a853eb2797409591e364f4bbc6582d"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "431d86b54d74d21c9534538f9e85a2ce"
server_key_iv: "f89107d4b45497a96cde9b8b624022d4"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.625747 18831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:07.642912 18831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:07.647142 18831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:07.647396 18831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:07.648052 18831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:07.649048 18831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:07.649116 18831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.649175 18831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:07.649207 18831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.658720 18831 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:44323
I20260808 08:20:07.658738 19026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:44323 every 8 connection(s)
I20260808 08:20:07.659474 18831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:07.669664 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 18831
I20260808 08:20:07.669782 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:07.670106 12340 external_mini_cluster.cc:1428] Setting key 6937ac9f675ef836bf1e79a5b4af88e4
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:07.674748 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.661445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:60881 (local address 127.12.13.62:42877)
0808 08:20:07.661610 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.661614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.662555 (+ 941us) server_negotiation.cc:407] Connection header received
0808 08:20:07.664053 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.664057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.664122 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.664243 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:07.665582 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.666254 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.667531 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.667778 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.667898 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.671018 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:07.671021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:07.671023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:07.671051 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:07.672581 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.673131 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.673134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.673137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.673220 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.673592 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.673596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.673597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.673799 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:07.673952 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.674467 (+ 515us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.674590 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:20:07.675621 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.661801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42877 (local address 127.12.13.2:60881)
0808 08:20:07.662383 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:07.662435 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:07.663815 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:07.664437 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:07.664445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:07.664852 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:07.665419 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.665436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.666408 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.666417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.667391 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.667401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.667963 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.667968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.668148 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.668903 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:07.668920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:07.670810 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:07.672722 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:07.672728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:07.672745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:07.673022 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:07.673314 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:07.673318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:07.673320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:07.673490 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:07.673947 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:07.673954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:07.674219 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 08:20:07.674424 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 08:20:07.674766 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":127,"threads_started":1}
I20260808 08:20:07.676879 19027 heartbeater.cc:347] Connected to a master server at 127.12.13.62:42877
I20260808 08:20:07.677140 19027 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:07.677616 19027 heartbeater.cc:511] Master 127.12.13.62:42877 requested a full tablet report, sending...
I20260808 08:20:07.678830 18536 ts_manager.cc:194] Registered new tserver with Master: e5a853eb2797409591e364f4bbc6582d (127.12.13.2:44323)
I20260808 08:20:07.679689 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:60881
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:20:07.701078 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42877
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:07.819599 19035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:07.819916 19035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:07.820011 19035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:07.824077 19035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:07.824194 19035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:07.824302 19035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:07.829933 19035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:42877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:07.831209 19035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:07.832310 19035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:07.842213 19046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:07.842244 19043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:07.842494 19035 server_base.cc:1034] running on GCE node
W20260808 08:20:07.842262 19042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:07.843214 19035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:07.843936 19035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:07.845139 19035 hybrid_clock.cc:648] HybridClock initialized: now 1786177207845098 us; error 55 us; skew 500 ppm
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:07.848143 19035 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:20:07.849042 19035 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:07.849143 19035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:07.849406 19035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:07.851616 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "e0d775565dc64df58abf21c8d192958b"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "08d920191a6cee2ad1ec3595f26b3455"
server_key_iv: "3c5174490012975396326d20c5bbb512"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.852206 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "e0d775565dc64df58abf21c8d192958b"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "08d920191a6cee2ad1ec3595f26b3455"
server_key_iv: "3c5174490012975396326d20c5bbb512"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.856178 19035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:07.858770 19057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.859982 19035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:07.860127 19035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "e0d775565dc64df58abf21c8d192958b"
format_stamp: "Formatted at 2026-08-08 08:20:07 on dist-test-slave-6ntk"
server_key: "08d920191a6cee2ad1ec3595f26b3455"
server_key_iv: "3c5174490012975396326d20c5bbb512"
server_key_version: "encryptionkey@0"
I20260808 08:20:07.860249 19035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:07.872097 19035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:07.875320 19035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:07.875527 19035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:07.876113 19035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:07.876948 19035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:07.877020 19035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.877091 19035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:07.877142 19035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:07.886314 19035 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:34817
I20260808 08:20:07.886350 19218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:34817 every 8 connection(s)
I20260808 08:20:07.886958 19035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:07.888266 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 19035
I20260808 08:20:07.888455 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:07.888795 12340 external_mini_cluster.cc:1428] Setting key 22f30a333046c400fbc61fbfd8411e7f
Aug 08 08:20:07 dist-test-slave-6ntk krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:07.901294 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.888860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:38001 (local address 127.12.13.62:42877)
0808 08:20:07.889999 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.890004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.890096 (+ 92us) server_negotiation.cc:407] Connection header received
0808 08:20:07.891740 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.891743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.891788 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.891861 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:07.892969 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.893490 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.894706 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.895050 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.895133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.897701 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:07.897709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:07.897711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:07.897736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:07.899231 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.899748 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.899753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.899755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.899837 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.900186 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.900191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.900192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.900386 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:07.900496 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.900994 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.901115 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046}
I20260808 08:20:07.902350 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.889318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42877 (local address 127.12.13.3:38001)
0808 08:20:07.889942 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:07.890003 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:07.891543 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:07.891983 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:07.891990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:07.892353 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:07.892839 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.892852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.893604 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.893607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.894591 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.894601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.895184 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.895187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.895390 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.895947 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:07.895962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:07.897530 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:07.899362 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:07.899368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:07.899383 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:07.899640 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:07.899938 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:07.899942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:07.899944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:07.900085 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:07.900494 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:07.900501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:07.900760 (+ 259us) client_negotiation.cc:769] Sending connection context
0808 08:20:07.900952 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 08:20:07.901269 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260808 08:20:07.903457 19219 heartbeater.cc:347] Connected to a master server at 127.12.13.62:42877
I20260808 08:20:07.903707 19219 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:07.904170 19219 heartbeater.cc:511] Master 127.12.13.62:42877 requested a full tablet report, sending...
I20260808 08:20:07.905191 18533 ts_manager.cc:194] Registered new tserver with Master: e0d775565dc64df58abf21c8d192958b (127.12.13.3:34817)
I20260808 08:20:07.905743 18533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:38001
I20260808 08:20:07.915022 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:20:07.924552 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.916990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.12.13.62:42877)
0808 08:20:07.917158 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.917163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.917292 (+ 129us) server_negotiation.cc:407] Connection header received
0808 08:20:07.917396 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.917399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.917443 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.917513 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:07.918297 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.918799 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.919608 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.919810 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.919905 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.921404 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:07.921407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:07.921409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:07.921430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:07.922747 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.923191 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.923205 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.923207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.923295 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:07.923589 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:07.923592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:07.923594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:07.923831 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:07.923945 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.924312 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.924429 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 08:20:07.927773 18533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:07.930943 18533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:07.946617 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.939542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:39949 (local address 127.0.0.1:33732)
0808 08:20:07.939954 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:07.940185 (+ 231us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:07.940339 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:07.941057 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:07.941063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:07.941089 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:07.941266 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.941273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.942883 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.942886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.944472 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.944484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.945982 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.945986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.946112 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.946160 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 08:20:07.946284 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 08:20:07.946427 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260808 08:20:07.946659 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.939730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.12.13.1:39949)
0808 08:20:07.940064 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.940071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.940506 (+ 435us) server_negotiation.cc:407] Connection header received
0808 08:20:07.940592 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.940599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.940831 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.941423 (+ 592us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:07.941577 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.942751 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.944625 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.945482 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.945620 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.945793 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.946348 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.946468 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":129,"threads_started":1}
I20260808 08:20:07.947024 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.940730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:34817 (local address 127.0.0.1:60914)
0808 08:20:07.941150 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:07.941164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:07.941248 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:07.941990 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:07.941993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:07.942013 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:07.942132 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.942139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.943262 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.943265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.944504 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.944533 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.946588 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.946592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.946708 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.946735 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:20:07.946794 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 08:20:07.946881 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":42,"thread_start_us":101,"threads_started":1}
I20260808 08:20:07.947688 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.941198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.12.13.3:34817)
0808 08:20:07.941584 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.941595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.941622 (+ 27us) server_negotiation.cc:407] Connection header received
0808 08:20:07.941683 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.941686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.941816 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.942004 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:07.942259 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.943108 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.945671 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.946412 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.947212 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.947387 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.947437 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.947529 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":207,"threads_started":1}
I20260808 08:20:07.947762 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.940359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:44323 (local address 127.0.0.1:39634)
0808 08:20:07.940803 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:07.940821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:07.940948 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:07.941593 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:07.941596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:07.941617 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:07.941716 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.941722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.943246 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.943253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.945490 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:07.945500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.947387 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.947391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:07.947494 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.947516 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:20:07.947565 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 08:20:07.947629 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":1068,"thread_start_us":110,"threads_started":1}
I20260808 08:20:07.948462 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:07.940556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.12.13.2:44323)
0808 08:20:07.941130 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:07.941141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:07.941164 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:20:07.941240 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:07.941245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:07.941439 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:07.941897 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:07.942048 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.943095 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.945602 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:07.947252 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:07.948033 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:07.948167 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:07.948214 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 08:20:07.948299 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":143,"threads_started":1}
I20260808 08:20:07.950170 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 1c35e5d30d4640739ede5847108de4b3 (DEFAULT_TABLE table=test-table [id=bb728770210c4d7c8685c170d8cfbaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:07.950249 19126 tablet_service.cc:1467] Processing CreateTablet for tablet 1c35e5d30d4640739ede5847108de4b3 (DEFAULT_TABLE table=test-table [id=bb728770210c4d7c8685c170d8cfbaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:07.950618 18928 tablet_service.cc:1467] Processing CreateTablet for tablet 1c35e5d30d4640739ede5847108de4b3 (DEFAULT_TABLE table=test-table [id=bb728770210c4d7c8685c170d8cfbaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:07.951385 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c35e5d30d4640739ede5847108de4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:07.951385 19126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c35e5d30d4640739ede5847108de4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:07.951709 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c35e5d30d4640739ede5847108de4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:07.957928 19247 tablet_bootstrap.cc:492] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Bootstrap starting.
I20260808 08:20:07.959084 19248 tablet_bootstrap.cc:492] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc: Bootstrap starting.
I20260808 08:20:07.960191 19247 tablet_bootstrap.cc:654] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:07.960675 19250 tablet_bootstrap.cc:492] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Bootstrap starting.
I20260808 08:20:07.960922 19248 tablet_bootstrap.cc:654] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:07.961596 19247 log.cc:824] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:07.961740 19248 log.cc:824] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:07.962821 19250 tablet_bootstrap.cc:654] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:07.963660 19247 tablet_bootstrap.cc:492] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: No bootstrap required, opened a new log
I20260808 08:20:07.963869 19248 tablet_bootstrap.cc:492] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc: No bootstrap required, opened a new log
I20260808 08:20:07.963917 19247 ts_tablet_manager.cc:1397] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:20:07.964061 19248 ts_tablet_manager.cc:1397] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 08:20:07.964161 19250 log.cc:824] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:07.966465 19250 tablet_bootstrap.cc:492] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: No bootstrap required, opened a new log
I20260808 08:20:07.966670 19250 ts_tablet_manager.cc:1397] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:20:07.967293 19247 raft_consensus.cc:348] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:07.967558 19247 raft_consensus.cc:374] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:07.967617 19247 raft_consensus.cc:729] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a853eb2797409591e364f4bbc6582d, State: Initialized, Role: FOLLOWER
I20260808 08:20:07.968132 19247 consensus_queue.cc:260] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:07.968541 19248 raft_consensus.cc:348] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:07.968786 19248 raft_consensus.cc:374] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:07.968848 19248 raft_consensus.cc:729] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb92f175625645df81f865323faadefc, State: Initialized, Role: FOLLOWER
I20260808 08:20:07.969199 19247 ts_tablet_manager.cc:1428] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 08:20:07.969340 19248 consensus_queue.cc:260] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:07.969609 19027 heartbeater.cc:503] Master 127.12.13.62:42877 was elected leader, sending a full tablet report...
I20260808 08:20:07.970073 18822 heartbeater.cc:503] Master 127.12.13.62:42877 was elected leader, sending a full tablet report...
I20260808 08:20:07.970434 19248 ts_tablet_manager.cc:1428] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:20:07.970863 19250 raft_consensus.cc:348] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:07.971180 19250 raft_consensus.cc:374] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:07.971243 19250 raft_consensus.cc:729] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d775565dc64df58abf21c8d192958b, State: Initialized, Role: FOLLOWER
I20260808 08:20:07.971676 19250 consensus_queue.cc:260] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:07.972354 19219 heartbeater.cc:503] Master 127.12.13.62:42877 was elected leader, sending a full tablet report...
I20260808 08:20:07.972748 19250 ts_tablet_manager.cc:1428] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 08:20:08.054798 19256 raft_consensus.cc:482] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:08.055065 19256 raft_consensus.cc:504] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:08.056430 19256 leader_election.cc:288] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb92f175625645df81f865323faadefc (127.12.13.1:39949), e0d775565dc64df58abf21c8d192958b (127.12.13.3:34817)
I20260808 08:20:08.060842 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.056689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:39949 (local address 127.12.13.2:39163)
0808 08:20:08.056845 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:08.056864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:08.056978 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:08.057379 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:08.057383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:08.057406 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:08.057560 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:08.057568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.058491 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.058494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:08.059490 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:08.059503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.060326 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.060332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:08.060455 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:08.060524 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 08:20:08.060613 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 08:20:08.060675 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":24}
I20260808 08:20:08.060884 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.056883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:39163 (local address 127.12.13.1:39949)
0808 08:20:08.057033 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:08.057042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:08.057061 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:20:08.057133 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:08.057135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:08.057176 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:08.057285 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:08.057692 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.058378 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.059636 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.060162 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.060300 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:08.060382 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:08.060642 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 08:20:08.060738 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:20:08.060995 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.057207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:38879 (local address 127.12.13.3:34817)
0808 08:20:08.057373 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:08.057378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:08.057531 (+ 153us) server_negotiation.cc:407] Connection header received
0808 08:20:08.057592 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:08.057598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:08.057664 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:08.057734 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:08.058064 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.058648 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.059750 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.060269 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.060387 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:08.060457 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:08.060788 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 08:20:08.060865 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 08:20:08.061015 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.057018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:34817 (local address 127.12.13.2:38879)
0808 08:20:08.057413 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:08.057435 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:08.057525 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:08.057772 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:08.057776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:08.057794 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:08.057952 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:08.057960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.058765 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.058767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:08.059603 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:08.059614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.060467 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.060471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:08.060643 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:08.060670 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:20:08.060746 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 08:20:08.060809 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":30,"thread_start_us":168,"threads_started":1}
I20260808 08:20:08.061695 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c35e5d30d4640739ede5847108de4b3" candidate_uuid: "e5a853eb2797409591e364f4bbc6582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb92f175625645df81f865323faadefc" is_pre_election: true
I20260808 08:20:08.061692 19156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c35e5d30d4640739ede5847108de4b3" candidate_uuid: "e5a853eb2797409591e364f4bbc6582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d775565dc64df58abf21c8d192958b" is_pre_election: true
I20260808 08:20:08.062062 18757 raft_consensus.cc:2454] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5a853eb2797409591e364f4bbc6582d in term 0.
I20260808 08:20:08.062068 19156 raft_consensus.cc:2454] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5a853eb2797409591e364f4bbc6582d in term 0.
I20260808 08:20:08.062568 18855 leader_election.cc:302] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d775565dc64df58abf21c8d192958b, e5a853eb2797409591e364f4bbc6582d; no voters:
I20260808 08:20:08.062846 19256 raft_consensus.cc:2781] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:08.062942 19256 raft_consensus.cc:482] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:08.062988 19256 raft_consensus.cc:3037] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:08.064203 19256 raft_consensus.cc:504] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:08.064558 19256 leader_election.cc:288] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [CANDIDATE]: Term 1 election: Requested vote from peers fb92f175625645df81f865323faadefc (127.12.13.1:39949), e0d775565dc64df58abf21c8d192958b (127.12.13.3:34817)
I20260808 08:20:08.064973 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c35e5d30d4640739ede5847108de4b3" candidate_uuid: "e5a853eb2797409591e364f4bbc6582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb92f175625645df81f865323faadefc"
I20260808 08:20:08.064973 19156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c35e5d30d4640739ede5847108de4b3" candidate_uuid: "e5a853eb2797409591e364f4bbc6582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d775565dc64df58abf21c8d192958b"
I20260808 08:20:08.065106 19156 raft_consensus.cc:3037] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:08.065105 18757 raft_consensus.cc:3037] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:08.066524 18757 raft_consensus.cc:2454] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5a853eb2797409591e364f4bbc6582d in term 1.
I20260808 08:20:08.066524 19156 raft_consensus.cc:2454] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5a853eb2797409591e364f4bbc6582d in term 1.
I20260808 08:20:08.066864 18855 leader_election.cc:302] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d775565dc64df58abf21c8d192958b, e5a853eb2797409591e364f4bbc6582d; no voters:
I20260808 08:20:08.067031 19256 raft_consensus.cc:2781] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:08.067262 19256 raft_consensus.cc:686] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 1 LEADER]: Becoming Leader. State: Replica: e5a853eb2797409591e364f4bbc6582d, State: Running, Role: LEADER
I20260808 08:20:08.067620 19256 consensus_queue.cc:237] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } }
I20260808 08:20:08.071071 18532 catalog_manager.cc:5495] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5a853eb2797409591e364f4bbc6582d (127.12.13.2). New cstate: current_term: 1 leader_uuid: "e5a853eb2797409591e364f4bbc6582d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5a853eb2797409591e364f4bbc6582d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:08.099160 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.095225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.12.13.2:44323)
0808 08:20:08.095393 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:08.095398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:08.095412 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:08.095561 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:08.095563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:08.095678 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:08.095815 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:08.096105 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.096630 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.097529 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:08.097754 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:08.097872 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:08.098241 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:08.098708 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:08.098859 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:08.098926 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 08:20:08.098987 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:20:08.105597 18757 raft_consensus.cc:1261] T 1c35e5d30d4640739ede5847108de4b3 P fb92f175625645df81f865323faadefc [term 1 FOLLOWER]: Refusing update from remote peer e5a853eb2797409591e364f4bbc6582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:08.105597 19156 raft_consensus.cc:1261] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 1 FOLLOWER]: Refusing update from remote peer e5a853eb2797409591e364f4bbc6582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:08.106487 19256 consensus_queue.cc:1035] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d775565dc64df58abf21c8d192958b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:08.106736 19265 consensus_queue.cc:1035] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb92f175625645df81f865323faadefc" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:08.115362 19269 mvcc.cc:205] Tried to move back new op lower bound from 7316181844392247296 to 7316181844247838720. Current Snapshot: MvccSnapshot[applied={T|T < 7316181844392247296}]
I20260808 08:20:08.115909 19270 mvcc.cc:205] Tried to move back new op lower bound from 7316181844392247296 to 7316181844247838720. Current Snapshot: MvccSnapshot[applied={T|T < 7316181844392247296}]
I20260808 08:20:08.118120 19271 mvcc.cc:205] Tried to move back new op lower bound from 7316181844392247296 to 7316181844247838720. Current Snapshot: MvccSnapshot[applied={T|T < 7316181844392247296}]
I20260808 08:20:08.129833 18532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:08.130085 18532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:08.133141 18532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eac29ff2fbee4bdfb2674632ffea3fee: Sending DeleteTablet for 3 replicas of tablet 1c35e5d30d4640739ede5847108de4b3
I20260808 08:20:08.134425 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c35e5d30d4640739ede5847108de4b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:08 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:33732
I20260808 08:20:08.134425 19126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c35e5d30d4640739ede5847108de4b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:08 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:60914
I20260808 08:20:08.134938 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 18629
I20260808 08:20:08.137805 18928 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c35e5d30d4640739ede5847108de4b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:08 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:39634
I20260808 08:20:08.141144 19286 tablet_replica.cc:330] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: stopping tablet replica
I20260808 08:20:08.141685 19286 raft_consensus.cc:2229] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:08.142375 19286 raft_consensus.cc:2258] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:08.143352 19283 tablet_replica.cc:330] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: stopping tablet replica
I20260808 08:20:08.144294 19283 raft_consensus.cc:2229] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:08.144821 19283 raft_consensus.cc:2258] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:08.148232 19286 ts_tablet_manager.cc:1905] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:08.148373 19283 ts_tablet_manager.cc:1905] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:08.150830 19283 ts_tablet_manager.cc:1918] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:08.150960 19283 log.cc:1196] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/1c35e5d30d4640739ede5847108de4b3
I20260808 08:20:08.151297 19283 ts_tablet_manager.cc:1939] T 1c35e5d30d4640739ede5847108de4b3 P e0d775565dc64df58abf21c8d192958b: Deleting consensus metadata
I20260808 08:20:08.151842 19286 ts_tablet_manager.cc:1918] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:08.151984 19286 log.cc:1196] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/1c35e5d30d4640739ede5847108de4b3
I20260808 08:20:08.152426 19286 ts_tablet_manager.cc:1939] T 1c35e5d30d4640739ede5847108de4b3 P e5a853eb2797409591e364f4bbc6582d: Deleting consensus metadata
I20260808 08:20:08.152767 18513 catalog_manager.cc:4841] TS e0d775565dc64df58abf21c8d192958b (127.12.13.3:34817): tablet 1c35e5d30d4640739ede5847108de4b3 (table test-table [id=bb728770210c4d7c8685c170d8cfbaa4]) successfully deleted
I20260808 08:20:08.153918 18513 catalog_manager.cc:4841] TS e5a853eb2797409591e364f4bbc6582d (127.12.13.2:44323): tablet 1c35e5d30d4640739ede5847108de4b3 (table test-table [id=bb728770210c4d7c8685c170d8cfbaa4]) successfully deleted
W20260808 08:20:08.161711 18513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:39949 (error 108)
I20260808 08:20:08.161840 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 18831
I20260808 08:20:08.165688 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.165299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:39949 (local address 127.0.0.1:33734)
0808 08:20:08.165482 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:08.165573 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 08:20:08.165983 18513 catalog_manager.cc:4584] TS fb92f175625645df81f865323faadefc (127.12.13.1:39949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c35e5d30d4640739ede5847108de4b3: Network error: Client connection negotiation failed: client connection to 127.12.13.1:39949: connect: Connection refused (error 111)
I20260808 08:20:08.174523 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 19035
I20260808 08:20:08.187502 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 18492
2026-08-08T08:20:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2697 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T08:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:08Z Disabled control of system clock
I20260808 08:20:08.232663 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36737
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 08:20:08.351904 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:08.352288 19291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:08.352370 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:08.356518 19291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:08.356637 19291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:08.356668 19291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:08.356710 19291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:08.356760 19291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:08.361820 19291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:08.363124 19291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:08.364651 19291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:08.373944 19304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:08.373944 19301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:08.373970 19302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:08.374012 19291 server_base.cc:1034] running on GCE node
I20260808 08:20:08.374889 19291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:08.376091 19291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:08.377331 19291 hybrid_clock.cc:648] HybridClock initialized: now 1786177208377311 us; error 34 us; skew 500 ppm
I20260808 08:20:08.379421 19291 webserver.cc:460] Webserver started at http://127.12.13.62:43739/ using document root <none> and password file <none>
I20260808 08:20:08.380056 19291 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:08.380149 19291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:08.380406 19291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:08.382592 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "9460b66d609d46dc831c34d25e6512b2"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "c47e80474e7c33f04d270284559724de"
server_key_iv: "052d2d5b6d02fe80fd6e031e1befc0c2"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.383167 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "9460b66d609d46dc831c34d25e6512b2"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "c47e80474e7c33f04d270284559724de"
server_key_iv: "052d2d5b6d02fe80fd6e031e1befc0c2"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.387240 19291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:08.389863 19314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.391237 19291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:08.391408 19291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "9460b66d609d46dc831c34d25e6512b2"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "c47e80474e7c33f04d270284559724de"
server_key_iv: "052d2d5b6d02fe80fd6e031e1befc0c2"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.391536 19291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:08.396405 19291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:08.397040 19291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:08.397225 19291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:08.404588 19291 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36737
I20260808 08:20:08.404596 19394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36737 every 8 connection(s)
I20260808 08:20:08.405581 19291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:08.409245 19395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:08.409586 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 19291
I20260808 08:20:08.409719 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:08.410068 12340 external_mini_cluster.cc:1428] Setting key ee54aa6d645619da670d28ae7fbd0ef4
I20260808 08:20:08.415915 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.411563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.12.13.62:36737)
0808 08:20:08.412286 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:08.412304 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:08.412339 (+ 35us) server_negotiation.cc:407] Connection header received
0808 08:20:08.413046 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:08.413064 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:08.413419 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:08.413764 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:08.414609 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:08.414614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:08.414626 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:08.414635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:08.414688 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:08.414703 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:08.414889 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:08.414968 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 08:20:08.415216 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":144,"threads_started":1}
I20260808 08:20:08.416889 19395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2: Bootstrap starting.
I20260808 08:20:08.419353 19395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:08.420310 19395 log.cc:824] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:08.422482 19395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2: No bootstrap required, opened a new log
I20260808 08:20:08.426419 19395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9460b66d609d46dc831c34d25e6512b2" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36737 } }
I20260808 08:20:08.426671 19395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:08.426707 19395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9460b66d609d46dc831c34d25e6512b2, State: Initialized, Role: FOLLOWER
I20260808 08:20:08.427105 19395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9460b66d609d46dc831c34d25e6512b2" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36737 } }
I20260808 08:20:08.427223 19395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:08.427271 19395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:08.427345 19395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:08.428303 19395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9460b66d609d46dc831c34d25e6512b2" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36737 } }
I20260808 08:20:08.428620 19395 leader_election.cc:302] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9460b66d609d46dc831c34d25e6512b2; no voters:
I20260808 08:20:08.428912 19395 leader_election.cc:288] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:08.429064 19401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:08.429324 19401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [term 1 LEADER]: Becoming Leader. State: Replica: 9460b66d609d46dc831c34d25e6512b2, State: Running, Role: LEADER
I20260808 08:20:08.429656 19401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9460b66d609d46dc831c34d25e6512b2" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36737 } }
I20260808 08:20:08.430339 19395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:08.431555 19402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9460b66d609d46dc831c34d25e6512b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9460b66d609d46dc831c34d25e6512b2" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36737 } } }
I20260808 08:20:08.431703 19402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:08.431602 19403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9460b66d609d46dc831c34d25e6512b2. Latest consensus state: current_term: 1 leader_uuid: "9460b66d609d46dc831c34d25e6512b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9460b66d609d46dc831c34d25e6512b2" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36737 } } }
I20260808 08:20:08.431762 19403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:08.432150 19413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:08.435729 19413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:08.442797 19413 catalog_manager.cc:1347] Generated new cluster ID: 0f01187028114c8890830b86c0481a35
I20260808 08:20:08.442888 19413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:08.462304 19413 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:08.462734 19413 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:08.471921 19413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2: Generated new TSK 0
I20260808 08:20:08.472774 19413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:08.478005 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36737
--builtin_ntp_servers=127.12.13.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 08:20:08.604087 19430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:08.604391 19430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:08.604434 19430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:08.608580 19430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:08.608646 19430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:08.608704 19430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:08.614130 19430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:08.615422 19430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:08.616952 19430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:08.628012 19436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:08.628012 19440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:08.628017 19438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:08.628186 19430 server_base.cc:1034] running on GCE node
I20260808 08:20:08.628724 19430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:08.629417 19430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:08.630625 19430 hybrid_clock.cc:648] HybridClock initialized: now 1786177208630606 us; error 30 us; skew 500 ppm
I20260808 08:20:08.632937 19430 webserver.cc:460] Webserver started at http://127.12.13.1:43775/ using document root <none> and password file <none>
I20260808 08:20:08.633598 19430 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:08.633701 19430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:08.634042 19430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:08.636298 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "5f07b75dd9984383ab62ee140b7d036b"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "76f93cec0083d0a0fc2155d0ede1f6a1"
server_key_iv: "675b05e5b2b0a9307f0d8f5198bb78b5"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.636930 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "5f07b75dd9984383ab62ee140b7d036b"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "76f93cec0083d0a0fc2155d0ede1f6a1"
server_key_iv: "675b05e5b2b0a9307f0d8f5198bb78b5"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.641261 19430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 08:20:08.644328 19448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.645955 19430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:08.646126 19430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "5f07b75dd9984383ab62ee140b7d036b"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "76f93cec0083d0a0fc2155d0ede1f6a1"
server_key_iv: "675b05e5b2b0a9307f0d8f5198bb78b5"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.646267 19430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:08.651676 19430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:08.652361 19430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:08.652565 19430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:08.653178 19430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:08.654263 19430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:08.654348 19430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.654420 19430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:08.654474 19430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.663467 19430 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:33439
I20260808 08:20:08.663488 19627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:33439 every 8 connection(s)
I20260808 08:20:08.664422 19430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:08.666172 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 19430
I20260808 08:20:08.666311 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:08.666694 12340 external_mini_cluster.cc:1428] Setting key 5cd316c62aa9fa8ad60b7ffac7cbdc8b
I20260808 08:20:08.669157 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36737
--builtin_ntp_servers=127.12.13.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 08:20:08.671803 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.666425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59861 (local address 127.12.13.62:36737)
0808 08:20:08.666590 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:08.666594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:08.667889 (+ 1295us) server_negotiation.cc:407] Connection header received
0808 08:20:08.669777 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:08.669779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:08.669819 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:08.669970 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:08.671301 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:08.671305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:08.671307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:08.671320 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:08.671342 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:08.671365 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:08.671464 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:08.671621 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 08:20:08.671667 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 08:20:08.672711 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.666919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36737 (local address 127.12.13.1:59861)
0808 08:20:08.667724 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:08.667766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:08.669535 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:08.670147 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:08.670158 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:08.670280 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:08.670907 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:08.670912 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:08.670976 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:08.671082 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:08.671450 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:08.671460 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:08.671466 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:20:08.671656 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 08:20:08.671923 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":167,"threads_started":1}
I20260808 08:20:08.674109 19629 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36737
I20260808 08:20:08.674355 19629 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:08.674784 19629 heartbeater.cc:511] Master 127.12.13.62:36737 requested a full tablet report, sending...
I20260808 08:20:08.676627 19337 ts_manager.cc:194] Registered new tserver with Master: 5f07b75dd9984383ab62ee140b7d036b (127.12.13.1:33439)
W20260808 08:20:08.790326 19634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:08.790617 19634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:08.790658 19634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:08.794660 19634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:08.794718 19634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:08.794772 19634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:08.799840 19634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:08.800977 19634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:08.802415 19634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:08.813007 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:08.812999 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:08.813202 19634 server_base.cc:1034] running on GCE node
W20260808 08:20:08.813251 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:08.814073 19634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:08.814849 19634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:08.816052 19634 hybrid_clock.cc:648] HybridClock initialized: now 1786177208816029 us; error 34 us; skew 500 ppm
I20260808 08:20:08.818123 19634 webserver.cc:460] Webserver started at http://127.12.13.2:40523/ using document root <none> and password file <none>
I20260808 08:20:08.818769 19634 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:08.818877 19634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:08.819137 19634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:08.821317 19634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "c20313b25ab2457d8885179365e9d564"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "8c179444093d7c1b455a6b4eed48a0c0"
server_key_iv: "be8c5bb3d0fb82dc4af867d3c4129c84"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.822068 19634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "c20313b25ab2457d8885179365e9d564"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "8c179444093d7c1b455a6b4eed48a0c0"
server_key_iv: "be8c5bb3d0fb82dc4af867d3c4129c84"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.826267 19634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:08.829120 19661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.830535 19634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:08.830706 19634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "c20313b25ab2457d8885179365e9d564"
format_stamp: "Formatted at 2026-08-08 08:20:08 on dist-test-slave-6ntk"
server_key: "8c179444093d7c1b455a6b4eed48a0c0"
server_key_iv: "be8c5bb3d0fb82dc4af867d3c4129c84"
server_key_version: "encryptionkey@0"
I20260808 08:20:08.830861 19634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:08.835948 19634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:08.836618 19634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:08.836818 19634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:08.837460 19634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:08.838543 19634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:08.838620 19634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.838696 19634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:08.838745 19634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:08.848337 19634 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:40755
I20260808 08:20:08.848364 19823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:40755 every 8 connection(s)
I20260808 08:20:08.849345 19634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:08.855692 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.851350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:56271 (local address 127.12.13.62:36737)
0808 08:20:08.851498 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:08.851502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:08.852625 (+ 1123us) server_negotiation.cc:407] Connection header received
0808 08:20:08.854031 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:08.854034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:08.854073 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:08.854151 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:08.855280 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:08.855282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:08.855284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:08.855297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:08.855318 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:08.855325 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:08.855407 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:08.855537 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 08:20:08.855576 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:20:08.856570 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:08.851774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36737 (local address 127.12.13.2:56271)
0808 08:20:08.852457 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:08.852500 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:08.853737 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:08.854317 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:08.854323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:08.854433 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:08.855017 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:08.855019 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:08.855062 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:08.855131 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:08.855405 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:08.855411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:08.855416 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 08:20:08.855553 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 08:20:08.855752 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260808 08:20:08.857723 19824 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36737
I20260808 08:20:08.858003 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 19634
I20260808 08:20:08.858037 19824 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:08.858139 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:08.858443 12340 external_mini_cluster.cc:1428] Setting key a63dbe6e231756316f704164c7628aea
I20260808 08:20:08.858541 19824 heartbeater.cc:511] Master 127.12.13.62:36737 requested a full tablet report, sending...
I20260808 08:20:08.859609 19336 ts_manager.cc:194] Registered new tserver with Master: c20313b25ab2457d8885179365e9d564 (127.12.13.2:40755)
I20260808 08:20:08.860598 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36737
--builtin_ntp_servers=127.12.13.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 08:20:08.979084 19829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:08.979378 19829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:08.979420 19829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:08.983594 19829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:08.983670 19829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:08.983728 19829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:08.988667 19829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:08.989797 19829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:08.991304 19829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:09.001413 19840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:09.001407 19838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:09.001535 19829 server_base.cc:1034] running on GCE node
W20260808 08:20:09.001729 19837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:09.002171 19829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:09.002879 19829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:09.004053 19829 hybrid_clock.cc:648] HybridClock initialized: now 1786177209004038 us; error 27 us; skew 500 ppm
I20260808 08:20:09.006107 19829 webserver.cc:460] Webserver started at http://127.12.13.3:36939/ using document root <none> and password file <none>
I20260808 08:20:09.006731 19829 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:09.006796 19829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:09.007062 19829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:09.009255 19829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "5d5d6fc4449b412d812f38a063a58913"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "7a005123cfea58cf91ca320f0a47ecfe"
server_key_iv: "067ca9d845442e69e28cd34956c77fcc"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.009779 19829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "5d5d6fc4449b412d812f38a063a58913"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "7a005123cfea58cf91ca320f0a47ecfe"
server_key_iv: "067ca9d845442e69e28cd34956c77fcc"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.013990 19829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:09.016714 19849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:09.018177 19829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:09.018330 19829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "5d5d6fc4449b412d812f38a063a58913"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "7a005123cfea58cf91ca320f0a47ecfe"
server_key_iv: "067ca9d845442e69e28cd34956c77fcc"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.018433 19829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:09.023293 19829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:09.023938 19829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:09.024145 19829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:09.024732 19829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:09.025724 19829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:09.025794 19829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:09.025946 19829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:09.025997 19829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:09.035998 19829 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:38847
I20260808 08:20:09.036017 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:38847 every 8 connection(s)
I20260808 08:20:09.037045 19829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:09.037962 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 19829
I20260808 08:20:09.038087 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:09.038384 12340 external_mini_cluster.cc:1428] Setting key 502a7b09e5c072e5bbe01825206dc6d4
I20260808 08:20:09.044188 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.039347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:50775 (local address 127.12.13.62:36737)
0808 08:20:09.039502 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.039508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.040864 (+ 1356us) server_negotiation.cc:407] Connection header received
0808 08:20:09.042367 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.042370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.042413 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.042492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.043770 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.043773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.043775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.043786 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.043807 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.043814 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.043901 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.044030 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.044072 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 08:20:09.045150 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.039899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36737 (local address 127.12.13.3:50775)
0808 08:20:09.040700 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.040761 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.042141 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.042677 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.042685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.042796 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:09.043498 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:09.043500 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:09.043546 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:09.043620 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:09.043903 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:09.043911 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:09.043915 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 08:20:09.044056 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 08:20:09.044268 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":153,"threads_started":1}
I20260808 08:20:09.046437 20019 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36737
I20260808 08:20:09.046670 20019 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:09.047083 20019 heartbeater.cc:511] Master 127.12.13.62:36737 requested a full tablet report, sending...
I20260808 08:20:09.048158 19336 ts_manager.cc:194] Registered new tserver with Master: 5d5d6fc4449b412d812f38a063a58913 (127.12.13.3:38847)
I20260808 08:20:09.052801 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:20:09.056588 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.054994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.12.13.62:36737)
0808 08:20:09.055126 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.055130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.055316 (+ 186us) server_negotiation.cc:407] Connection header received
0808 08:20:09.055447 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.055449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.055484 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.055553 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.056307 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.056309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.056311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.056319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.056333 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.056340 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.056405 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.056448 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.056480 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:20:09.059999 19336 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:09.063337 19336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:09.075842 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.072234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:38847 (local address 127.0.0.1:56004)
0808 08:20:09.072633 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.072694 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.072814 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.074101 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.074106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.074171 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:09.074907 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:09.074909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:09.074926 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:09.075021 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:09.075517 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:09.075523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:09.075529 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:20:09.075612 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 08:20:09.075661 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
I20260808 08:20:09.075897 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.072949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.12.13.1:33439)
0808 08:20:09.073820 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.073832 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.073856 (+ 24us) server_negotiation.cc:407] Connection header received
0808 08:20:09.073943 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.073948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.074117 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.074252 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.075182 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.075194 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.075204 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.075214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.075253 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.075277 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.075403 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.075635 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.075716 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":136,"threads_started":1}
I20260808 08:20:09.075842 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.072510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:33439 (local address 127.0.0.1:52994)
0808 08:20:09.073210 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.073223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.073305 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.074251 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.074254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.074266 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:09.074993 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:09.074995 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:09.075003 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:09.075020 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:09.075400 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:09.075412 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:09.075421 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 08:20:09.075566 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 08:20:09.075675 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":21,"thread_start_us":121,"threads_started":1}
I20260808 08:20:09.076761 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.072394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.12.13.3:38847)
0808 08:20:09.073567 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.073580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.073605 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:20:09.073665 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.073672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.073934 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.074080 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.075334 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.075340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.075352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.075361 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.075400 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.075423 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.075537 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.076463 (+ 926us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.076554 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":118,"threads_started":1}
I20260808 08:20:09.077657 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.073055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40755 (local address 127.0.0.1:45318)
0808 08:20:09.075033 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.075050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.075137 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.076417 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.076422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.076441 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:09.076989 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:09.076993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:09.077006 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:09.077039 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:09.077405 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:09.077409 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:09.077412 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 08:20:09.077504 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 08:20:09.077540 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1890,"thread_start_us":101,"threads_started":1}
I20260808 08:20:09.077777 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.073378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.12.13.2:40755)
0808 08:20:09.073736 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.073752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.075233 (+ 1481us) server_negotiation.cc:407] Connection header received
0808 08:20:09.075296 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.075301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.075495 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.075625 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.077191 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.077199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.077210 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.077223 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.077267 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.077299 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.077409 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.077545 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.077617 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
I20260808 08:20:09.079403 19530 tablet_service.cc:1467] Processing CreateTablet for tablet a5329f5c120d4a90bfdf1bd97b2afd18 (DEFAULT_TABLE table=test-table [id=607beb99e8dd42048279ba111bee5f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:09.079731 19930 tablet_service.cc:1467] Processing CreateTablet for tablet a5329f5c120d4a90bfdf1bd97b2afd18 (DEFAULT_TABLE table=test-table [id=607beb99e8dd42048279ba111bee5f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:09.080703 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5329f5c120d4a90bfdf1bd97b2afd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:09.080807 19930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5329f5c120d4a90bfdf1bd97b2afd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:09.080464 19732 tablet_service.cc:1467] Processing CreateTablet for tablet a5329f5c120d4a90bfdf1bd97b2afd18 (DEFAULT_TABLE table=test-table [id=607beb99e8dd42048279ba111bee5f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:09.081413 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5329f5c120d4a90bfdf1bd97b2afd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:09.087535 20045 tablet_bootstrap.cc:492] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Bootstrap starting.
I20260808 08:20:09.089097 20046 tablet_bootstrap.cc:492] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Bootstrap starting.
I20260808 08:20:09.090198 20045 tablet_bootstrap.cc:654] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:09.090437 20052 tablet_bootstrap.cc:492] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b: Bootstrap starting.
I20260808 08:20:09.091135 20045 log.cc:824] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:09.091466 20046 tablet_bootstrap.cc:654] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:09.092275 20052 tablet_bootstrap.cc:654] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:09.092371 20046 log.cc:824] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:09.093063 20052 log.cc:824] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:09.093640 20045 tablet_bootstrap.cc:492] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: No bootstrap required, opened a new log
I20260808 08:20:09.093940 20045 ts_tablet_manager.cc:1397] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 08:20:09.094435 20046 tablet_bootstrap.cc:492] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: No bootstrap required, opened a new log
I20260808 08:20:09.094676 20046 ts_tablet_manager.cc:1397] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:20:09.095248 20052 tablet_bootstrap.cc:492] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b: No bootstrap required, opened a new log
I20260808 08:20:09.095433 20052 ts_tablet_manager.cc:1397] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 08:20:09.098620 20046 raft_consensus.cc:348] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.098905 20046 raft_consensus.cc:374] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:09.098974 20046 raft_consensus.cc:729] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20313b25ab2457d8885179365e9d564, State: Initialized, Role: FOLLOWER
I20260808 08:20:09.098973 20045 raft_consensus.cc:348] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.099237 20045 raft_consensus.cc:374] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:09.099309 20045 raft_consensus.cc:729] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5d6fc4449b412d812f38a063a58913, State: Initialized, Role: FOLLOWER
I20260808 08:20:09.099503 20046 consensus_queue.cc:260] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.099772 20045 consensus_queue.cc:260] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.100392 19824 heartbeater.cc:503] Master 127.12.13.62:36737 was elected leader, sending a full tablet report...
I20260808 08:20:09.100205 20052 raft_consensus.cc:348] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.100474 20052 raft_consensus.cc:374] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:09.100528 20052 raft_consensus.cc:729] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f07b75dd9984383ab62ee140b7d036b, State: Initialized, Role: FOLLOWER
I20260808 08:20:09.100853 20046 ts_tablet_manager.cc:1428] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 08:20:09.100942 20052 consensus_queue.cc:260] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.101655 19629 heartbeater.cc:503] Master 127.12.13.62:36737 was elected leader, sending a full tablet report...
I20260808 08:20:09.101693 20045 ts_tablet_manager.cc:1428] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 08:20:09.101959 20052 ts_tablet_manager.cc:1428] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260808 08:20:09.102030 20019 heartbeater.cc:503] Master 127.12.13.62:36737 was elected leader, sending a full tablet report...
I20260808 08:20:09.112092 20061 raft_consensus.cc:482] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:09.112344 20061 raft_consensus.cc:504] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.113566 20061 leader_election.cc:288] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20313b25ab2457d8885179365e9d564 (127.12.13.2:40755), 5f07b75dd9984383ab62ee140b7d036b (127.12.13.1:33439)
I20260808 08:20:09.115854 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.114214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:38635 (local address 127.12.13.2:40755)
0808 08:20:09.114344 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.114366 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.114384 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:20:09.114430 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.114433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.114472 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.114571 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.115430 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.115433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.115435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.115447 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.115469 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.115477 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.115593 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.115652 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.115701 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:20:09.115872 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.113938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40755 (local address 127.12.13.3:38635)
0808 08:20:09.114045 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.114063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.114140 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.114618 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.114624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.114641 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:09.115239 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:09.115241 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:09.115251 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:09.115275 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:09.115580 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:09.115582 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:09.115584 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 08:20:09.115650 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 08:20:09.115703 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 08:20:09.115927 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.113938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:33439 (local address 127.12.13.3:40613)
0808 08:20:09.114529 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.114546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.114639 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.114907 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.114912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.114929 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:09.115422 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:09.115424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:09.115431 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:09.115450 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:09.115703 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:09.115706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:09.115708 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 08:20:09.115782 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 08:20:09.115809 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":250,"thread_start_us":121,"threads_started":1}
I20260808 08:20:09.115948 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.113990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:40613 (local address 127.12.13.1:33439)
0808 08:20:09.114131 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.114135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.114636 (+ 501us) server_negotiation.cc:407] Connection header received
0808 08:20:09.114719 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.114722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.114765 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.114854 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.115566 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.115569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.115571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.115583 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.115606 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.115618 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.115707 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.115777 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.115813 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 08:20:09.116580 19764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5329f5c120d4a90bfdf1bd97b2afd18" candidate_uuid: "5d5d6fc4449b412d812f38a063a58913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20313b25ab2457d8885179365e9d564" is_pre_election: true
I20260808 08:20:09.116732 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5329f5c120d4a90bfdf1bd97b2afd18" candidate_uuid: "5d5d6fc4449b412d812f38a063a58913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f07b75dd9984383ab62ee140b7d036b" is_pre_election: true
I20260808 08:20:09.116865 19764 raft_consensus.cc:2454] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5d6fc4449b412d812f38a063a58913 in term 0.
I20260808 08:20:09.117055 19563 raft_consensus.cc:2454] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5d6fc4449b412d812f38a063a58913 in term 0.
I20260808 08:20:09.117357 19852 leader_election.cc:302] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5d6fc4449b412d812f38a063a58913, c20313b25ab2457d8885179365e9d564; no voters:
I20260808 08:20:09.117700 20061 raft_consensus.cc:2781] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:09.117798 20061 raft_consensus.cc:482] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:09.117906 20061 raft_consensus.cc:3037] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:09.119012 20061 raft_consensus.cc:504] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.119374 20061 leader_election.cc:288] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [CANDIDATE]: Term 1 election: Requested vote from peers c20313b25ab2457d8885179365e9d564 (127.12.13.2:40755), 5f07b75dd9984383ab62ee140b7d036b (127.12.13.1:33439)
I20260808 08:20:09.119742 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5329f5c120d4a90bfdf1bd97b2afd18" candidate_uuid: "5d5d6fc4449b412d812f38a063a58913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f07b75dd9984383ab62ee140b7d036b"
I20260808 08:20:09.119740 19764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5329f5c120d4a90bfdf1bd97b2afd18" candidate_uuid: "5d5d6fc4449b412d812f38a063a58913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20313b25ab2457d8885179365e9d564"
I20260808 08:20:09.119887 19563 raft_consensus.cc:3037] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:09.119907 19764 raft_consensus.cc:3037] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:09.121002 19563 raft_consensus.cc:2454] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5d6fc4449b412d812f38a063a58913 in term 1.
I20260808 08:20:09.121052 19764 raft_consensus.cc:2454] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5d6fc4449b412d812f38a063a58913 in term 1.
I20260808 08:20:09.121344 19854 leader_election.cc:302] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5d6fc4449b412d812f38a063a58913, 5f07b75dd9984383ab62ee140b7d036b; no voters:
I20260808 08:20:09.121522 20061 raft_consensus.cc:2781] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:09.121820 20061 raft_consensus.cc:686] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 1 LEADER]: Becoming Leader. State: Replica: 5d5d6fc4449b412d812f38a063a58913, State: Running, Role: LEADER
I20260808 08:20:09.122222 20061 consensus_queue.cc:237] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } }
I20260808 08:20:09.126035 19336 catalog_manager.cc:5495] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d5d6fc4449b412d812f38a063a58913 (127.12.13.3). New cstate: current_term: 1 leader_uuid: "5d5d6fc4449b412d812f38a063a58913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5d6fc4449b412d812f38a063a58913" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:09.143021 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.141566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.12.13.3:38847)
0808 08:20:09.141701 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.141705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.141719 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:09.141806 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.141808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.141930 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.142017 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.142676 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.142679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.142681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.142693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.142714 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.142723 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.142797 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.142871 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.142907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:20:09.149490 19764 raft_consensus.cc:1261] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 1 FOLLOWER]: Refusing update from remote peer 5d5d6fc4449b412d812f38a063a58913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:09.149490 19563 raft_consensus.cc:1261] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5f07b75dd9984383ab62ee140b7d036b [term 1 FOLLOWER]: Refusing update from remote peer 5d5d6fc4449b412d812f38a063a58913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:09.150305 20061 consensus_queue.cc:1035] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20313b25ab2457d8885179365e9d564" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:09.150554 20064 consensus_queue.cc:1035] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f07b75dd9984383ab62ee140b7d036b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:09.158809 20075 mvcc.cc:205] Tried to move back new op lower bound from 7316181848668299264 to 7316181848567873536. Current Snapshot: MvccSnapshot[applied={T|T < 7316181848668299264}]
I20260808 08:20:09.167882 19336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:09.168088 19336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:09.170800 19336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9460b66d609d46dc831c34d25e6512b2: Sending DeleteTablet for 3 replicas of tablet a5329f5c120d4a90bfdf1bd97b2afd18
I20260808 08:20:09.171800 19530 tablet_service.cc:1514] Processing DeleteTablet for tablet a5329f5c120d4a90bfdf1bd97b2afd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:09 UTC) from {username='slave'} at 127.0.0.1:52994
I20260808 08:20:09.171819 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet a5329f5c120d4a90bfdf1bd97b2afd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:09 UTC) from {username='slave'} at 127.0.0.1:45318
I20260808 08:20:09.172379 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 19430
I20260808 08:20:09.172559 19930 tablet_service.cc:1514] Processing DeleteTablet for tablet a5329f5c120d4a90bfdf1bd97b2afd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:09 UTC) from {username='slave'} at 127.0.0.1:56004
I20260808 08:20:09.175168 20090 tablet_replica.cc:330] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: stopping tablet replica
I20260808 08:20:09.177225 20090 raft_consensus.cc:2229] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:09.177796 20090 raft_consensus.cc:2258] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:09.181671 20090 ts_tablet_manager.cc:1905] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:09.187515 20090 ts_tablet_manager.cc:1918] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:09.187754 20090 log.cc:1196] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/a5329f5c120d4a90bfdf1bd97b2afd18
I20260808 08:20:09.188278 20090 ts_tablet_manager.cc:1939] T a5329f5c120d4a90bfdf1bd97b2afd18 P c20313b25ab2457d8885179365e9d564: Deleting consensus metadata
I20260808 08:20:09.194941 19320 catalog_manager.cc:4841] TS c20313b25ab2457d8885179365e9d564 (127.12.13.2:40755): tablet a5329f5c120d4a90bfdf1bd97b2afd18 (table test-table [id=607beb99e8dd42048279ba111bee5f07]) successfully deleted
I20260808 08:20:09.199293 20091 tablet_replica.cc:330] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: stopping tablet replica
I20260808 08:20:09.199721 20091 raft_consensus.cc:2229] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:09.200208 20091 raft_consensus.cc:2258] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:09.204882 20091 ts_tablet_manager.cc:1905] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:09.208034 20091 ts_tablet_manager.cc:1918] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:09.208181 20091 log.cc:1196] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/a5329f5c120d4a90bfdf1bd97b2afd18
I20260808 08:20:09.208614 20091 ts_tablet_manager.cc:1939] T a5329f5c120d4a90bfdf1bd97b2afd18 P 5d5d6fc4449b412d812f38a063a58913: Deleting consensus metadata
I20260808 08:20:09.210124 19320 catalog_manager.cc:4841] TS 5d5d6fc4449b412d812f38a063a58913 (127.12.13.3:38847): tablet a5329f5c120d4a90bfdf1bd97b2afd18 (table test-table [id=607beb99e8dd42048279ba111bee5f07]) successfully deleted
W20260808 08:20:09.211004 19321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:33439 (error 108)
I20260808 08:20:09.211769 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 19634
I20260808 08:20:09.219928 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.211925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:33439 (local address 127.0.0.1:52998)
0808 08:20:09.219629 (+ 7704us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.219771 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7544}
W20260808 08:20:09.222147 19321 catalog_manager.cc:4584] TS 5f07b75dd9984383ab62ee140b7d036b (127.12.13.1:33439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5329f5c120d4a90bfdf1bd97b2afd18: Network error: Client connection negotiation failed: client connection to 127.12.13.1:33439: connect: Connection refused (error 111)
I20260808 08:20:09.227990 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 19829
I20260808 08:20:09.243554 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 19291
2026-08-08T08:20:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1053 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 08:20:09.275792 12340 mini_oidc.cc:152] Starting JWKS server
I20260808 08:20:09.277667 12340 webserver.cc:460] Webserver started at https://127.0.0.1:46467/ using document root <none> and password file <none>
I20260808 08:20:09.277822 12340 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 08:20:09.278169 12340 webserver.cc:460] Webserver started at http://127.0.0.1:36405/ using document root <none> and password file <none>
2026-08-08T08:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:09Z Disabled control of system clock
I20260808 08:20:09.290139 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:39063
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:39063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46467/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/testchainca.cert with env {}
W20260808 08:20:09.415791 20103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:09.416115 20103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:09.416157 20103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:09.420392 20103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:09.420471 20103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:09.420487 20103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:09.420504 20103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:09.420521 20103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 08:20:09.420537 20103 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 08:20:09.420559 20103 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46467/jwks.json
I20260808 08:20:09.426362 20103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:39063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46467/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:09.427659 20103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:09.428730 20103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:09.438057 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:09.438047 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:09.438050 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:09.438112 20103 server_base.cc:1034] running on GCE node
I20260808 08:20:09.439016 20103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:09.440265 20103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:09.441458 20103 hybrid_clock.cc:648] HybridClock initialized: now 1786177209441423 us; error 49 us; skew 500 ppm
I20260808 08:20:09.444567 20103 webserver.cc:460] Webserver started at http://127.12.13.62:37913/ using document root <none> and password file <none>
I20260808 08:20:09.445323 20103 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:09.445396 20103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:09.445582 20103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:09.447965 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "eacba3bbccd646caa276971d66b9152a"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "91df5f8d205c60eef9f5c7b22eab74a4"
server_key_iv: "efc0c6364f4d0804bae6fc8adafd73d6"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.448562 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "eacba3bbccd646caa276971d66b9152a"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "91df5f8d205c60eef9f5c7b22eab74a4"
server_key_iv: "efc0c6364f4d0804bae6fc8adafd73d6"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.453027 20103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:20:09.455911 20121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:09.457350 20103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:09.457533 20103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "eacba3bbccd646caa276971d66b9152a"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "91df5f8d205c60eef9f5c7b22eab74a4"
server_key_iv: "efc0c6364f4d0804bae6fc8adafd73d6"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.457677 20103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:09.494168 20103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:09.495112 20103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:09.495370 20103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:09.503829 20103 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:39063
I20260808 08:20:09.503862 20200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:39063 every 8 connection(s)
I20260808 08:20:09.504874 20103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:09.508584 20201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:09.508699 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 20103
I20260808 08:20:09.508868 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:09.509177 12340 external_mini_cluster.cc:1428] Setting key bbf575a70a764ac4d3dfed9804815e8e
I20260808 08:20:09.516229 20201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a: Bootstrap starting.
I20260808 08:20:09.518924 20201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:09.519835 20201 log.cc:824] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:09.520466 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.510595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.12.13.62:39063)
0808 08:20:09.511266 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.511288 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.511326 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:20:09.512170 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.512190 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.512603 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.512903 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.513259 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.514400 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.515650 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.516345 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.516513 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:09.517934 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.517941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.517958 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.518456 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.518528 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.518555 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.518655 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.519139 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.519436 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":154,"threads_started":1}
I20260808 08:20:09.523027 20201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a: No bootstrap required, opened a new log
I20260808 08:20:09.525434 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:20:09.528012 20201 raft_consensus.cc:348] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacba3bbccd646caa276971d66b9152a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39063 } }
I20260808 08:20:09.528270 20201 raft_consensus.cc:374] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:09.528360 20201 raft_consensus.cc:729] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacba3bbccd646caa276971d66b9152a, State: Initialized, Role: FOLLOWER
I20260808 08:20:09.528841 20201 consensus_queue.cc:260] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacba3bbccd646caa276971d66b9152a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39063 } }
I20260808 08:20:09.528993 20201 raft_consensus.cc:388] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:09.529083 20201 raft_consensus.cc:482] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:09.529186 20201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:09.530210 20201 raft_consensus.cc:504] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacba3bbccd646caa276971d66b9152a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39063 } }
I20260808 08:20:09.530612 20201 leader_election.cc:302] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eacba3bbccd646caa276971d66b9152a; no voters:
I20260808 08:20:09.530977 20201 leader_election.cc:288] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:09.531090 20207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:09.531358 20207 raft_consensus.cc:686] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [term 1 LEADER]: Becoming Leader. State: Replica: eacba3bbccd646caa276971d66b9152a, State: Running, Role: LEADER
I20260808 08:20:09.531667 20207 consensus_queue.cc:237] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacba3bbccd646caa276971d66b9152a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39063 } }
I20260808 08:20:09.532476 20201 sys_catalog.cc:564] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:09.533493 20208 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eacba3bbccd646caa276971d66b9152a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacba3bbccd646caa276971d66b9152a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39063 } } }
I20260808 08:20:09.533658 20208 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:09.533975 20209 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eacba3bbccd646caa276971d66b9152a. Latest consensus state: current_term: 1 leader_uuid: "eacba3bbccd646caa276971d66b9152a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacba3bbccd646caa276971d66b9152a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39063 } } }
I20260808 08:20:09.534065 20218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:09.534093 20209 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:09.537106 20218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:09.543625 20218 catalog_manager.cc:1347] Generated new cluster ID: dbafd9b6689b4325ba64fb564518c70e
I20260808 08:20:09.543717 20218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:09.563840 20218 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:09.565212 20218 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:09.583477 20218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eacba3bbccd646caa276971d66b9152a: Generated new TSK 0
I20260808 08:20:09.584424 20218 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 08:20:09.626135 20245 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 08:20:09.626327 20245 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 08:20:09.626905 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.599361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.12.13.62:39063)
0808 08:20:09.599487 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.599492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.599813 (+ 321us) server_negotiation.cc:407] Connection header received
0808 08:20:09.600090 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.600094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.600146 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.600242 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0808 08:20:09.600552 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.601103 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.602424 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.602643 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.602726 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:09.603128 (+ 402us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 08:20:09.626359 (+ 23231us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 08:20:09.626743 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 08:20:09.627020 20204 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 08:20:09.628398 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:09.599188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:39063 (local address 127.0.0.1:38698)
0808 08:20:09.599680 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.599696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.599893 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.600285 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.600288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.600297 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 08:20:09.600439 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:09.600447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.601215 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.601219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:09.602271 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:09.602283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.602761 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.602765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:09.602870 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:09.602926 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 08:20:09.626681 (+ 23755us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 08:20:09.626978 (+ 297us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.13.62:39063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.13.62:39063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 08:20:09.629223 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 20103
2026-08-08T08:20:09Z chronyd exiting
I20260808 08:20:09.680433 12340 test_util.cc:175] -----------------------------------------------
I20260808 08:20:09.680532 12340 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786177179845590-12340-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (406 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 08:20:09.681705 12340 mini_oidc.cc:152] Starting JWKS server
I20260808 08:20:09.682685 12340 webserver.cc:460] Webserver started at https://127.0.0.1:43305/ using document root <none> and password file <none>
I20260808 08:20:09.682801 12340 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 08:20:09.683095 12340 webserver.cc:460] Webserver started at http://127.0.0.1:34171/ using document root <none> and password file <none>
2026-08-08T08:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:09Z Disabled control of system clock
I20260808 08:20:09.694528 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:41111
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:41111
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43305/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/testchainca.cert with env {}
W20260808 08:20:09.815536 20254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:09.815828 20254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:09.815878 20254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:09.819906 20254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:09.819979 20254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:09.819994 20254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:09.820050 20254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:09.820080 20254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 08:20:09.820106 20254 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 08:20:09.820135 20254 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43305/jwks.json
I20260808 08:20:09.825841 20254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:41111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43305/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:09.827055 20254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:09.828094 20254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:09.837195 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:09.837464 20254 server_base.cc:1034] running on GCE node
W20260808 08:20:09.837244 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:09.837193 20263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:09.838249 20254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:09.839462 20254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:09.840668 20254 hybrid_clock.cc:648] HybridClock initialized: now 1786177209840646 us; error 35 us; skew 500 ppm
I20260808 08:20:09.843065 20254 webserver.cc:460] Webserver started at http://127.12.13.62:33077/ using document root <none> and password file <none>
I20260808 08:20:09.843784 20254 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:09.843858 20254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:09.844117 20254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:09.846706 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "4a93aa721afc40f3bdb4b379bfedc59d"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "0ca5ef2e71a1aff157350a95f368b0ec"
server_key_iv: "e9d3bec8a94e15469caf0c3dc9241ba4"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.847401 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "4a93aa721afc40f3bdb4b379bfedc59d"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "0ca5ef2e71a1aff157350a95f368b0ec"
server_key_iv: "e9d3bec8a94e15469caf0c3dc9241ba4"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.852114 20254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:09.855170 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:09.856748 20254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:09.856951 20254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "4a93aa721afc40f3bdb4b379bfedc59d"
format_stamp: "Formatted at 2026-08-08 08:20:09 on dist-test-slave-6ntk"
server_key: "0ca5ef2e71a1aff157350a95f368b0ec"
server_key_iv: "e9d3bec8a94e15469caf0c3dc9241ba4"
server_key_version: "encryptionkey@0"
I20260808 08:20:09.857107 20254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:09.882687 20254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:09.883561 20254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:09.883793 20254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:09.891337 20254 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:41111
I20260808 08:20:09.891403 20364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:41111 every 8 connection(s)
I20260808 08:20:09.892453 20254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:09.895912 20365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:09.901809 20365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d: Bootstrap starting.
I20260808 08:20:09.901818 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 20254
I20260808 08:20:09.901971 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:09.902258 12340 external_mini_cluster.cc:1428] Setting key 268fc5045b8b85db7d1f20bfd9429ac6
I20260808 08:20:09.904695 20365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:09.905484 20365 log.cc:824] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:09.907689 20365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d: No bootstrap required, opened a new log
I20260808 08:20:09.911764 20365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41111 } }
I20260808 08:20:09.912009 20365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:09.912081 20365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a93aa721afc40f3bdb4b379bfedc59d, State: Initialized, Role: FOLLOWER
I20260808 08:20:09.912616 20365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41111 } }
I20260808 08:20:09.912672 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.903653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.12.13.62:41111)
0808 08:20:09.904424 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.904450 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.904505 (+ 55us) server_negotiation.cc:407] Connection header received
0808 08:20:09.905251 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.905271 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.905782 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.906268 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:09.906724 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.907716 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.908748 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.909272 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.909502 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:09.910549 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:09.910554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:09.910568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:09.910996 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:09.911059 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:09.911081 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:09.911207 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:09.911383 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 08:20:09.911671 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":157,"threads_started":1}
I20260808 08:20:09.912746 20365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:09.912824 20365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:09.912923 20365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:09.914032 20365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41111 } }
I20260808 08:20:09.914412 20365 leader_election.cc:302] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a93aa721afc40f3bdb4b379bfedc59d; no voters:
I20260808 08:20:09.914799 20365 leader_election.cc:288] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:09.914975 20371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:09.915210 20371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [term 1 LEADER]: Becoming Leader. State: Replica: 4a93aa721afc40f3bdb4b379bfedc59d, State: Running, Role: LEADER
I20260808 08:20:09.915575 20371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41111 } }
I20260808 08:20:09.915972 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:20:09.916249 20365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:09.917618 20372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41111 } } }
I20260808 08:20:09.917621 20374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a93aa721afc40f3bdb4b379bfedc59d. Latest consensus state: current_term: 1 leader_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a93aa721afc40f3bdb4b379bfedc59d" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 41111 } } }
I20260808 08:20:09.917764 20372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:09.917766 20374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:09.918495 20382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:09.921911 20382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:09.927604 20382 catalog_manager.cc:1347] Generated new cluster ID: a2f639aaf2454fbea261757c30e5755a
I20260808 08:20:09.927688 20382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:09.933718 20382 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:09.934743 20382 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:09.945541 20382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a93aa721afc40f3bdb4b379bfedc59d: Generated new TSK 0
I20260808 08:20:09.946429 20382 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 08:20:09.983980 20407 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 08:20:09.984086 20407 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 08:20:09.984843 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:09.961327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.12.13.62:41111)
0808 08:20:09.961579 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:09.961584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:09.961758 (+ 174us) server_negotiation.cc:407] Connection header received
0808 08:20:09.961983 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:09.961986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:09.962039 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:09.962137 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0808 08:20:09.962468 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.963009 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.964337 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.964574 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.964664 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:09.965081 (+ 417us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 08:20:09.984456 (+ 19375us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43305/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 08:20:09.984709 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43305/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":166}
W20260808 08:20:09.984949 20369 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43305/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 08:20:09.985172 20406 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:09.961170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:41111 (local address 127.0.0.1:37470)
0808 08:20:09.961633 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:09.961655 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:09.961802 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:09.962177 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:09.962181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:09.962191 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 08:20:09.962338 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:09.962346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.963154 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.963158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:09.964175 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:09.964188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:09.964716 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:09.964720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:09.964819 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:09.964854 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 08:20:09.984705 (+ 19851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43305/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 08:20:09.984951 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.13.62:41111: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43305/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":121,"threads_started":1}
I20260808 08:20:09.985864 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 20254
2026-08-08T08:20:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 08:20:10.031167 12340 mini_oidc.cc:152] Starting JWKS server
I20260808 08:20:10.032112 12340 webserver.cc:460] Webserver started at https://127.0.0.1:41571/ using document root <none> and password file <none>
I20260808 08:20:10.032253 12340 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 08:20:10.032534 12340 webserver.cc:460] Webserver started at http://127.0.0.1:36335/ using document root <none> and password file <none>
2026-08-08T08:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:10Z Disabled control of system clock
I20260808 08:20:10.043560 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35689
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:35689
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41571/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 08:20:10.165587 20417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:10.166033 20417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:10.166105 20417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:10.170182 20417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:10.170238 20417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:10.170254 20417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:10.170271 20417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:10.170286 20417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 08:20:10.170302 20417 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 08:20:10.170326 20417 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41571/jwks.json
I20260808 08:20:10.175208 20417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:35689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41571/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:10.176308 20417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:10.177690 20417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:10.186473 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:10.186480 20426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:10.186730 20417 server_base.cc:1034] running on GCE node
W20260808 08:20:10.186916 20427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:10.187582 20417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:10.188761 20417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:10.189999 20417 hybrid_clock.cc:648] HybridClock initialized: now 1786177210189980 us; error 41 us; skew 500 ppm
I20260808 08:20:10.192746 20417 webserver.cc:460] Webserver started at http://127.12.13.62:42137/ using document root <none> and password file <none>
I20260808 08:20:10.193500 20417 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:10.193570 20417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:10.193868 20417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:10.196780 20417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "e564ed175be5451d8abd969cebc3093e"
format_stamp: "Formatted at 2026-08-08 08:20:10 on dist-test-slave-6ntk"
server_key: "8a4674b58fdfb210487911cc7e505bc9"
server_key_iv: "46fc653e3fd16b9b15c009efbed0c488"
server_key_version: "encryptionkey@0"
I20260808 08:20:10.197396 20417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "e564ed175be5451d8abd969cebc3093e"
format_stamp: "Formatted at 2026-08-08 08:20:10 on dist-test-slave-6ntk"
server_key: "8a4674b58fdfb210487911cc7e505bc9"
server_key_iv: "46fc653e3fd16b9b15c009efbed0c488"
server_key_version: "encryptionkey@0"
I20260808 08:20:10.201575 20417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260808 08:20:10.204247 20437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:10.205685 20417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:10.205844 20417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "e564ed175be5451d8abd969cebc3093e"
format_stamp: "Formatted at 2026-08-08 08:20:10 on dist-test-slave-6ntk"
server_key: "8a4674b58fdfb210487911cc7e505bc9"
server_key_iv: "46fc653e3fd16b9b15c009efbed0c488"
server_key_version: "encryptionkey@0"
I20260808 08:20:10.206034 20417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:10.230715 20417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:10.231545 20417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:10.231770 20417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:10.239506 20417 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:35689
I20260808 08:20:10.239518 20517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:35689 every 8 connection(s)
I20260808 08:20:10.240640 20417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:10.244539 20518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:10.250558 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 20417
I20260808 08:20:10.250681 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:10.250908 20518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e: Bootstrap starting.
I20260808 08:20:10.251019 12340 external_mini_cluster.cc:1428] Setting key a06c5e9fa5f5983a62533be6547a71e3
I20260808 08:20:10.253834 20518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:10.254673 20518 log.cc:824] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:10.257164 20518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e: No bootstrap required, opened a new log
I20260808 08:20:10.262251 20518 raft_consensus.cc:348] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e564ed175be5451d8abd969cebc3093e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35689 } }
I20260808 08:20:10.262564 20518 raft_consensus.cc:374] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:10.262629 20518 raft_consensus.cc:729] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e564ed175be5451d8abd969cebc3093e, State: Initialized, Role: FOLLOWER
I20260808 08:20:10.263008 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:10.252603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.12.13.62:35689)
0808 08:20:10.253401 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:10.253418 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:10.253457 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:20:10.254338 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:10.254362 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:10.254866 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:10.255277 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:10.255765 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.257376 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.258458 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.259260 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.259463 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:10.260336 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:10.260345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:10.260366 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:10.261034 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:10.261133 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:10.261169 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:10.261292 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:10.261538 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 08:20:10.261893 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":204,"threads_started":1}
I20260808 08:20:10.263172 20518 consensus_queue.cc:260] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e564ed175be5451d8abd969cebc3093e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35689 } }
I20260808 08:20:10.263391 20518 raft_consensus.cc:388] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:10.263466 20518 raft_consensus.cc:482] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:10.263576 20518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:10.264788 20518 raft_consensus.cc:504] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e564ed175be5451d8abd969cebc3093e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35689 } }
I20260808 08:20:10.265219 20518 leader_election.cc:302] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e564ed175be5451d8abd969cebc3093e; no voters:
I20260808 08:20:10.265645 20518 leader_election.cc:288] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:10.265746 20527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:10.266033 20527 raft_consensus.cc:686] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [term 1 LEADER]: Becoming Leader. State: Replica: e564ed175be5451d8abd969cebc3093e, State: Running, Role: LEADER
I20260808 08:20:10.266278 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:20:10.266388 20527 consensus_queue.cc:237] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e564ed175be5451d8abd969cebc3093e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35689 } }
I20260808 08:20:10.267110 20518 sys_catalog.cc:564] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:10.267699 20527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e564ed175be5451d8abd969cebc3093e. Latest consensus state: current_term: 1 leader_uuid: "e564ed175be5451d8abd969cebc3093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e564ed175be5451d8abd969cebc3093e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35689 } } }
I20260808 08:20:10.267818 20527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:10.268311 20528 sys_catalog.cc:455] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e564ed175be5451d8abd969cebc3093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e564ed175be5451d8abd969cebc3093e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35689 } } }
I20260808 08:20:10.268412 20532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:10.268522 20528 sys_catalog.cc:458] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:10.271946 20532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:10.278488 20532 catalog_manager.cc:1347] Generated new cluster ID: 7ed2d6250072484ab4ef842e57117255
I20260808 08:20:10.278589 20532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:10.287223 20532 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:10.288229 20532 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:10.293227 20532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e564ed175be5451d8abd969cebc3093e: Generated new TSK 0
I20260808 08:20:10.294018 20532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:10.316901 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:10.311113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.12.13.62:35689)
0808 08:20:10.311246 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:10.311252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:10.311530 (+ 278us) server_negotiation.cc:407] Connection header received
0808 08:20:10.311688 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:10.311691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:10.311802 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:10.311928 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0808 08:20:10.312142 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.313010 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.314163 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.314519 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.314626 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:10.314950 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 08:20:10.316489 (+ 1539us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 08:20:10.316743 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 08:20:10.316987 20525 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 08:20:10.318656 20559 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:10.311068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35689 (local address 127.0.0.1:58714)
0808 08:20:10.311430 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:10.311444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:10.311549 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:10.311931 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:10.311933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:10.311940 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 08:20:10.312036 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:10.312041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.313250 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.313256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:10.314032 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:10.314043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.314663 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.314666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:10.314738 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:10.314763 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 08:20:10.316667 (+ 1904us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 08:20:10.318482 (+ 1815us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.13.62:35689: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1587,"thread_start_us":166,"threads_started":1}
I20260808 08:20:10.319353 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 20417
E20260808 08:20:10.341254 20561 webserver.cc:533] Webserver: sslize failed:
E20260808 08:20:10.341351 20561 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T08:20:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T08:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:10Z Disabled control of system clock
I20260808 08:20:10.383383 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42099
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:42099
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/testchainca.cert with env {}
W20260808 08:20:10.502832 20568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:10.503201 20568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:10.503285 20568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:10.507535 20568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:10.507637 20568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:10.507664 20568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:10.507706 20568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:10.507759 20568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 08:20:10.507827 20568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 08:20:10.507880 20568 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 08:20:10.514344 20568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:42099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:10.515570 20568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:10.516706 20568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:10.526017 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:10.525977 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:10.525977 20579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:10.526270 20568 server_base.cc:1034] running on GCE node
I20260808 08:20:10.527048 20568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:10.528295 20568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:10.529498 20568 hybrid_clock.cc:648] HybridClock initialized: now 1786177210529488 us; error 32 us; skew 500 ppm
I20260808 08:20:10.531533 20568 webserver.cc:460] Webserver started at http://127.12.13.62:35611/ using document root <none> and password file <none>
I20260808 08:20:10.532166 20568 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:10.532264 20568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:10.532507 20568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:10.534771 20568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "cfaeb6a4e58f46b0afb12378fcce4127"
format_stamp: "Formatted at 2026-08-08 08:20:10 on dist-test-slave-6ntk"
server_key: "8be34b7a7749e6550b9e012f6bd34eca"
server_key_iv: "65be35e960dd1893de151ed2496c0c5c"
server_key_version: "encryptionkey@0"
I20260808 08:20:10.535367 20568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "cfaeb6a4e58f46b0afb12378fcce4127"
format_stamp: "Formatted at 2026-08-08 08:20:10 on dist-test-slave-6ntk"
server_key: "8be34b7a7749e6550b9e012f6bd34eca"
server_key_iv: "65be35e960dd1893de151ed2496c0c5c"
server_key_version: "encryptionkey@0"
I20260808 08:20:10.539425 20568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:10.542232 20588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:10.543665 20568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:20:10.543846 20568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "cfaeb6a4e58f46b0afb12378fcce4127"
format_stamp: "Formatted at 2026-08-08 08:20:10 on dist-test-slave-6ntk"
server_key: "8be34b7a7749e6550b9e012f6bd34eca"
server_key_iv: "65be35e960dd1893de151ed2496c0c5c"
server_key_version: "encryptionkey@0"
I20260808 08:20:10.543982 20568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:10.570600 20568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:10.571421 20568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:10.571656 20568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:10.579804 20568 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:42099
I20260808 08:20:10.579819 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:42099 every 8 connection(s)
I20260808 08:20:10.580852 20568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:10.584091 20666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:10.590013 20666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127: Bootstrap starting.
I20260808 08:20:10.590152 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 20568
I20260808 08:20:10.590301 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:10.590581 12340 external_mini_cluster.cc:1428] Setting key a1c961505d63cc7f21b42b0541f964e0
I20260808 08:20:10.592782 20666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:10.593583 20666 log.cc:824] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:10.595633 20666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127: No bootstrap required, opened a new log
I20260808 08:20:10.599586 20666 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42099 } }
I20260808 08:20:10.599848 20666 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:10.599946 20666 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfaeb6a4e58f46b0afb12378fcce4127, State: Initialized, Role: FOLLOWER
I20260808 08:20:10.600132 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:10.592032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.12.13.62:42099)
0808 08:20:10.592678 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:10.592692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:10.592727 (+ 35us) server_negotiation.cc:407] Connection header received
0808 08:20:10.593411 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:10.593429 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:10.593940 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:10.594277 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:10.594715 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.595666 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.596615 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.597131 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.597310 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:10.598242 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:10.598247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:10.598260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:10.598675 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:10.598732 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:10.598753 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:10.598864 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:10.599026 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 08:20:10.599287 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":146,"threads_started":1}
I20260808 08:20:10.600500 20666 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42099 } }
I20260808 08:20:10.600637 20666 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:10.600736 20666 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:10.600868 20666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:10.601811 20666 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42099 } }
I20260808 08:20:10.602196 20666 leader_election.cc:302] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfaeb6a4e58f46b0afb12378fcce4127; no voters:
I20260808 08:20:10.602561 20666 leader_election.cc:288] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:10.602640 20673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:10.602856 20673 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [term 1 LEADER]: Becoming Leader. State: Replica: cfaeb6a4e58f46b0afb12378fcce4127, State: Running, Role: LEADER
I20260808 08:20:10.603147 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:20:10.603168 20673 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42099 } }
I20260808 08:20:10.603945 20666 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:10.604434 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfaeb6a4e58f46b0afb12378fcce4127. Latest consensus state: current_term: 1 leader_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42099 } } }
I20260808 08:20:10.604550 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:10.605193 20674 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaeb6a4e58f46b0afb12378fcce4127" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42099 } } }
I20260808 08:20:10.605301 20674 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:10.606112 20678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:10.608925 20678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:10.614567 20678 catalog_manager.cc:1347] Generated new cluster ID: 6e78745ac22549ecaa40d89ef022e414
I20260808 08:20:10.614637 20678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:10.634672 20678 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:10.635690 20678 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:10.642634 20678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfaeb6a4e58f46b0afb12378fcce4127: Generated new TSK 0
I20260808 08:20:10.643426 20678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:11.108055 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:10.655527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.12.13.62:42099)
0808 08:20:10.655671 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:10.655676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:10.655885 (+ 209us) server_negotiation.cc:407] Connection header received
0808 08:20:10.656047 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:10.656050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:10.656101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:10.656194 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0808 08:20:10.656464 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.656998 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.658238 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.658458 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.658534 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:10.658898 (+ 364us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 08:20:11.107632 (+448734us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 08:20:11.107918 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39982: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 08:20:11.108170 20671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39982: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 08:20:11.108301 20709 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:10.655453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:42099 (local address 127.0.0.1:39982)
0808 08:20:10.655778 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:10.655794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:10.655918 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:10.656220 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:10.656223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:10.656231 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 08:20:10.656356 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:10.656363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.657106 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.657110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:10.658124 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:10.658136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:10.658581 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:10.658585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:10.658681 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:10.658719 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 08:20:11.107889 (+449170us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 08:20:11.108101 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.13.62:42099: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260808 08:20:11.108968 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 20568
2026-08-08T08:20:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (761 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (137 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (134 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:11 dist-test-slave-6ntk krb5kdc[20729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:11 dist-test-slave-6ntk krb5kdc[20729](info): set up 2 sockets
Aug 08 08:20:11 dist-test-slave-6ntk krb5kdc[20729](info): commencing operation
krb5kdc: starting...
W20260808 08:20:12.641291 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:12 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:12.777509 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36219
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:12.894086 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:12.894403 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:12.894486 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:12.898576 20743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:12.898665 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:12.898699 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:12.898754 20743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:12.898792 20743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:12.904094 20743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:12.905252 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:12.906499 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:20:12.915521 20743 server_base.cc:1034] running on GCE node
W20260808 08:20:12.915472 20750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:12.915469 20749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:12.915469 20753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:12.916318 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:12.917433 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:12.918599 20743 hybrid_clock.cc:648] HybridClock initialized: now 1786177212918578 us; error 32 us; skew 500 ppm
Aug 08 08:20:12 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:12.925046 20743 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:12.926153 20743 webserver.cc:460] Webserver started at http://127.12.13.62:43881/ using document root <none> and password file <none>
I20260808 08:20:12.926815 20743 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:12.926887 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:12.927126 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:12.929271 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "127d98bde2ef4b809509f58fae769be8"
format_stamp: "Formatted at 2026-08-08 08:20:12 on dist-test-slave-6ntk"
server_key: "923ade331903a99109c2aec8261dac5c"
server_key_iv: "a8feedccdc80c037a7e857b7b2d7a3f9"
server_key_version: "encryptionkey@0"
I20260808 08:20:12.929939 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "127d98bde2ef4b809509f58fae769be8"
format_stamp: "Formatted at 2026-08-08 08:20:12 on dist-test-slave-6ntk"
server_key: "923ade331903a99109c2aec8261dac5c"
server_key_iv: "a8feedccdc80c037a7e857b7b2d7a3f9"
server_key_version: "encryptionkey@0"
I20260808 08:20:12.933923 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 08:20:12.936501 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:12.937846 20743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:12.938066 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "127d98bde2ef4b809509f58fae769be8"
format_stamp: "Formatted at 2026-08-08 08:20:12 on dist-test-slave-6ntk"
server_key: "923ade331903a99109c2aec8261dac5c"
server_key_iv: "a8feedccdc80c037a7e857b7b2d7a3f9"
server_key_version: "encryptionkey@0"
I20260808 08:20:12.938189 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:12.955214 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:12.958323 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:12.958540 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:12.966162 20743 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36219
I20260808 08:20:12.966187 20841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36219 every 8 connection(s)
I20260808 08:20:12.967183 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:12.970024 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:12.974845 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 20743
I20260808 08:20:12.974972 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:12.975292 12340 external_mini_cluster.cc:1428] Setting key b810f419332983bb23e884e20c378676
I20260808 08:20:12.976034 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8: Bootstrap starting.
I20260808 08:20:12.978513 20843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:12.979270 20843 log.cc:824] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:12.981417 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8: No bootstrap required, opened a new log
Aug 08 08:20:12 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:12.985715 20843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127d98bde2ef4b809509f58fae769be8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36219 } }
I20260808 08:20:12.986085 20843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:12.986153 20843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127d98bde2ef4b809509f58fae769be8, State: Initialized, Role: FOLLOWER
I20260808 08:20:12.986712 20843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127d98bde2ef4b809509f58fae769be8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36219 } }
I20260808 08:20:12.986887 20843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:12.986970 20843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:12.987074 20843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:12.988212 20843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127d98bde2ef4b809509f58fae769be8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36219 } }
I20260808 08:20:12.988603 20843 leader_election.cc:302] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127d98bde2ef4b809509f58fae769be8; no voters:
I20260808 08:20:12.988989 20843 leader_election.cc:288] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:12.989109 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:12.989454 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [term 1 LEADER]: Becoming Leader. State: Replica: 127d98bde2ef4b809509f58fae769be8, State: Running, Role: LEADER
I20260808 08:20:12.990047 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127d98bde2ef4b809509f58fae769be8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36219 } }
I20260808 08:20:12.990432 20843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:12.990718 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:12.976779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.12.13.62:36219)
0808 08:20:12.977236 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:12.977253 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:12.977291 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:20:12.977960 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:12.977978 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:12.978310 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:12.978646 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:12.979528 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:12.980866 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:12.981769 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:12.982460 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:12.982636 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:12.985117 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:12.985124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:12.985141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:12.985173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:12.987416 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:12.987842 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:12.987850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:12.987857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:12.988014 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:12.988288 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:12.988291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:12.988293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:12.988829 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:12.989023 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:12.989375 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 08:20:12.989668 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":138,"threads_started":1}
I20260808 08:20:12.991981 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127d98bde2ef4b809509f58fae769be8. Latest consensus state: current_term: 1 leader_uuid: "127d98bde2ef4b809509f58fae769be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127d98bde2ef4b809509f58fae769be8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36219 } } }
I20260808 08:20:12.991984 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127d98bde2ef4b809509f58fae769be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127d98bde2ef4b809509f58fae769be8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36219 } } }
I20260808 08:20:12.992137 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:12.992137 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8 [sys.catalog]: This master's current role is: LEADER
W20260808 08:20:12.995227 20873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:20:12.995292 20873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:20:12.995380 20870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:12.996217 20870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:13.001796 20870 catalog_manager.cc:1347] Generated new cluster ID: f480defed13d48daa0dbb85cc865f5c0
I20260808 08:20:13.001904 20870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:13.010159 20870 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:13.011413 20870 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:13.016894 20870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127d98bde2ef4b809509f58fae769be8: Generated new TSK 0
I20260808 08:20:13.017688 20870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:20:13.059607 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36219
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:13.184500 20882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:13.184836 20882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:13.184932 20882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:13.189301 20882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:13.189419 20882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:13.189525 20882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:13.195219 20882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:13.196399 20882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:13.197547 20882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:13.208221 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:13.208221 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:13.208225 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:13.208489 20882 server_base.cc:1034] running on GCE node
I20260808 08:20:13.209111 20882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:13.209924 20882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:13.211143 20882 hybrid_clock.cc:648] HybridClock initialized: now 1786177213211113 us; error 47 us; skew 500 ppm
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.214279 20882 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:20:13.215229 20882 webserver.cc:460] Webserver started at http://127.12.13.1:37529/ using document root <none> and password file <none>
I20260808 08:20:13.215865 20882 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:13.215969 20882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:13.216219 20882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:13.218448 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "93f9f3f4947f473ebb791cb81f393f80"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "e4d2d1fdc03e2b5a2216a789765246df"
server_key_iv: "790fcde2abd1595bbfd8715874244016"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.219043 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "93f9f3f4947f473ebb791cb81f393f80"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "e4d2d1fdc03e2b5a2216a789765246df"
server_key_iv: "790fcde2abd1595bbfd8715874244016"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.223150 20882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:13.225653 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.226979 20882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:13.227126 20882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "93f9f3f4947f473ebb791cb81f393f80"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "e4d2d1fdc03e2b5a2216a789765246df"
server_key_iv: "790fcde2abd1595bbfd8715874244016"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.227247 20882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:13.248782 20882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:13.252530 20882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:13.252759 20882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:13.253561 20882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:13.254601 20882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:13.254652 20882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.254694 20882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:13.254709 20882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.264753 20882 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:35431
I20260808 08:20:13.264778 21085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:35431 every 8 connection(s)
I20260808 08:20:13.265730 20882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:13.268244 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 20882
I20260808 08:20:13.268402 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:13.268697 12340 external_mini_cluster.cc:1428] Setting key cef8fbd7ea140170083c8da35c786cf5
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:13.280694 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.267829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:49031 (local address 127.12.13.62:36219)
0808 08:20:13.267987 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.267994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.268771 (+ 777us) server_negotiation.cc:407] Connection header received
0808 08:20:13.270705 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.270708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.270766 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.270846 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.272083 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.272633 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.273828 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.274122 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.274238 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.276848 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:13.276852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:13.276855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:13.276883 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:13.278505 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.279046 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.279049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.279051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.279138 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.279506 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.279511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.279512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.279712 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:13.279825 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:13.280372 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 08:20:13.280506 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:20:13.281529 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.268109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36219 (local address 127.12.13.1:49031)
0808 08:20:13.268601 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:13.268658 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:13.270481 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:13.271032 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:13.271039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:13.271413 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:13.271921 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:13.271935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.272771 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.272774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:13.273685 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:13.273695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.274281 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.274286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:13.274464 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.275106 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:13.275124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:13.276675 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:13.278630 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:13.278636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:13.278652 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:13.278932 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:13.279236 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:13.279240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:13.279242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:13.279400 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:13.279825 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:13.279857 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:13.280116 (+ 259us) client_negotiation.cc:769] Sending connection context
0808 08:20:13.280319 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 08:20:13.280659 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:20:13.282879 21088 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36219
I20260808 08:20:13.283159 21088 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:13.283706 21088 heartbeater.cc:511] Master 127.12.13.62:36219 requested a full tablet report, sending...
I20260808 08:20:13.285341 20791 ts_manager.cc:194] Registered new tserver with Master: 93f9f3f4947f473ebb791cb81f393f80 (127.12.13.1:35431)
I20260808 08:20:13.286590 20791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:49031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:20:13.301791 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36219
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:13.422832 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:13.423095 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:13.423164 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:13.427273 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:13.427337 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:13.427423 21095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:13.432896 21095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:13.434056 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:13.435135 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:13.445309 21106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:13.445317 21104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:13.445604 21103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:13.445636 21095 server_base.cc:1034] running on GCE node
I20260808 08:20:13.446257 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:13.447001 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:13.448221 21095 hybrid_clock.cc:648] HybridClock initialized: now 1786177213448222 us; error 57 us; skew 500 ppm
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.451428 21095 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:20:13.452412 21095 webserver.cc:460] Webserver started at http://127.12.13.2:46417/ using document root <none> and password file <none>
I20260808 08:20:13.453022 21095 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:13.453086 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:13.453269 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:13.455611 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "555f73ab0612443b8f5bb871bd59ab97"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "90cd96235b633786cb0c388f0ab6898e"
server_key_iv: "66c0ec9093b071c2ef1c1b66ffcec4d4"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.456173 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "555f73ab0612443b8f5bb871bd59ab97"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "90cd96235b633786cb0c388f0ab6898e"
server_key_iv: "66c0ec9093b071c2ef1c1b66ffcec4d4"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.461416 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 08:20:13.464094 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.465452 21095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:13.465572 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "555f73ab0612443b8f5bb871bd59ab97"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "90cd96235b633786cb0c388f0ab6898e"
server_key_iv: "66c0ec9093b071c2ef1c1b66ffcec4d4"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.465668 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:13.482872 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:13.485807 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:13.486065 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:13.486645 21095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:13.487471 21095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:13.487519 21095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.487560 21095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:13.487574 21095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.497077 21095 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:34643
I20260808 08:20:13.497172 21283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:34643 every 8 connection(s)
I20260808 08:20:13.498090 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:13.499917 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 21095
I20260808 08:20:13.500041 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:13.500319 12340 external_mini_cluster.cc:1428] Setting key bae7bc0971491dace12612a5209ca3a4
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:20:13.514025 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.499966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:42225 (local address 127.12.13.62:36219)
0808 08:20:13.500127 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.500132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.501150 (+ 1018us) server_negotiation.cc:407] Connection header received
0808 08:20:13.502992 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.502994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.503041 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.503116 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.504639 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.505501 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.506715 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.507153 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.507282 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.510100 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:13.510104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:13.510110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:13.510143 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:13.511803 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.512397 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.512403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.512407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.512502 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.512818 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.512820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.512821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.513071 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:13.513154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:13.513704 (+ 550us) server_negotiation.cc:299] Negotiation successful
0808 08:20:13.513818 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:20:13.515024 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.500441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36219 (local address 127.12.13.2:42225)
0808 08:20:13.500976 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:13.501035 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:13.502741 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:13.503316 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:13.503329 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:13.503865 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:13.504491 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:13.504504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.505629 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.505632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:13.506586 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:13.506595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.507338 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.507343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:13.507501 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.508125 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:13.508141 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:13.509918 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:13.511965 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:13.511973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:13.511991 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:13.512252 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:13.512593 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:13.512595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:13.512596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:13.512726 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:13.513180 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:13.513189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:13.513475 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 08:20:13.513634 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 08:20:13.513965 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:20:13.516374 21284 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36219
I20260808 08:20:13.516734 21284 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:13.517385 21284 heartbeater.cc:511] Master 127.12.13.62:36219 requested a full tablet report, sending...
I20260808 08:20:13.518618 20791 ts_manager.cc:194] Registered new tserver with Master: 555f73ab0612443b8f5bb871bd59ab97 (127.12.13.2:34643)
I20260808 08:20:13.519212 20791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:42225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:20:13.534482 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36219
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:13.651880 21290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:13.652189 21290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:13.652242 21290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:13.656581 21290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:13.656665 21290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:13.656729 21290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:13.662271 21290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:13.663383 21290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:13.664433 21290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:13.674868 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:13.674937 21303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:13.674868 21302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:13.675273 21290 server_base.cc:1034] running on GCE node
I20260808 08:20:13.675695 21290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:13.676380 21290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:13.677616 21290 hybrid_clock.cc:648] HybridClock initialized: now 1786177213677597 us; error 31 us; skew 500 ppm
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.680611 21290 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:20:13.681564 21290 webserver.cc:460] Webserver started at http://127.12.13.3:41007/ using document root <none> and password file <none>
I20260808 08:20:13.682265 21290 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:13.682341 21290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:13.682566 21290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:13.684604 21290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "934c69a610494b659b93ac1bfac749b6"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "754b9be995af1a18267728ab17c4e419"
server_key_iv: "a2ab5ee994323f79116226b3173d924d"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.685254 21290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "934c69a610494b659b93ac1bfac749b6"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "754b9be995af1a18267728ab17c4e419"
server_key_iv: "a2ab5ee994323f79116226b3173d924d"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.689239 21290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260808 08:20:13.692085 21316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.693353 21290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 08:20:13.693480 21290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "934c69a610494b659b93ac1bfac749b6"
format_stamp: "Formatted at 2026-08-08 08:20:13 on dist-test-slave-6ntk"
server_key: "754b9be995af1a18267728ab17c4e419"
server_key_iv: "a2ab5ee994323f79116226b3173d924d"
server_key_version: "encryptionkey@0"
I20260808 08:20:13.693585 21290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:13.705756 21290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:13.708801 21290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:13.709010 21290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:13.709632 21290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:13.710548 21290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:13.710597 21290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.710669 21290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:13.710711 21290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:13.722307 21290 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:38763
I20260808 08:20:13.722309 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:38763 every 8 connection(s)
I20260808 08:20:13.723322 21290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:13.732676 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 21290
I20260808 08:20:13.732817 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:13.733158 12340 external_mini_cluster.cc:1428] Setting key 5f61b1c3bf8530320c5d02813deece33
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:13.737521 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.725262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:50899 (local address 127.12.13.62:36219)
0808 08:20:13.725482 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.725486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.726354 (+ 868us) server_negotiation.cc:407] Connection header received
0808 08:20:13.727695 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.727698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.727748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.727829 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.728984 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.729549 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.730679 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.730933 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.731034 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.733766 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:13.733770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:13.733772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:13.733803 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:13.735469 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.735981 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.735985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.735987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.736077 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.736428 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.736433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.736435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.736630 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:13.736739 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:13.737233 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 08:20:13.737359 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260808 08:20:13.738441 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.725663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36219 (local address 127.12.13.3:50899)
0808 08:20:13.726190 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:13.726234 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:13.727487 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:13.728004 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:13.728010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:13.728366 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:13.728857 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:13.728871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.729658 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.729660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:13.730555 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:13.730564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.731111 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.731115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:13.731263 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.731912 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:13.731929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:13.733632 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:13.735593 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:13.735599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:13.735614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:13.735876 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:13.736169 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:13.736173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:13.736175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:13.736319 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:13.736737 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:13.736744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:13.736996 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 08:20:13.737193 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 08:20:13.737538 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":197,"threads_started":1}
I20260808 08:20:13.739750 21476 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36219
I20260808 08:20:13.740010 21476 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:13.740590 21476 heartbeater.cc:511] Master 127.12.13.62:36219 requested a full tablet report, sending...
I20260808 08:20:13.741722 20791 ts_manager.cc:194] Registered new tserver with Master: 934c69a610494b659b93ac1bfac749b6 (127.12.13.3:38763)
I20260808 08:20:13.742460 20791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:50899
I20260808 08:20:13.747771 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/test-admin.keytab.
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.767887 12340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 08:20:13.770974 21489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:13.769335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.12.13.1:35431)
0808 08:20:13.769704 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.769720 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.769746 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:20:13.769801 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.769807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.770124 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.770252 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.770722 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35042: BlockingRecv error: recv got EOF from 127.0.0.1:35042 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":151,"threads_started":1}
W20260808 08:20:13.771226 12340 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc)
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.772316 12340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.773276 12340 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:20:13.783375 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.774232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.12.13.1:35431)
0808 08:20:13.774368 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.774373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.774497 (+ 124us) server_negotiation.cc:407] Connection header received
0808 08:20:13.774663 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.774666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.774707 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.774793 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.775365 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.776145 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.777011 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.777391 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.777509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.779923 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:13.779932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:13.779946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:13.779969 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:13.781458 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.781943 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.781950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.781951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.782059 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.782373 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.782377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.782379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.782643 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:13.782886 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:13.783064 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 08:20:13.783224 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 08:20:13.787726 21489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:13.786643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.12.13.1:35431)
0808 08:20:13.786773 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.786776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.786846 (+ 70us) server_negotiation.cc:407] Connection header received
0808 08:20:13.786949 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.786951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.786987 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.787052 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.787623 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35070: BlockingRecv error: recv got EOF from 127.0.0.1:35070 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 08:20:13.787881 12340 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc)
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.788961 12340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.789956 12340 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:20:13.799167 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.790869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.12.13.1:35431)
0808 08:20:13.791008 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.791012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.791038 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:20:13.791163 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.791165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.791213 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.791285 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.791944 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.792558 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.793314 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.793530 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.793606 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.795949 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:13.795952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:13.795954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:13.795976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:13.797568 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.798017 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.798019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.798021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.798095 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.798420 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.798423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.798424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.798584 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:13.798692 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:13.798863 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 08:20:13.798957 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 08:20:13.801782 21489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:13.800975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.12.13.1:35431)
0808 08:20:13.801071 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.801074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.801129 (+ 55us) server_negotiation.cc:407] Connection header received
0808 08:20:13.801223 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.801225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.801256 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.801320 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.801672 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35086: BlockingRecv error: recv got EOF from 127.0.0.1:35086 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260808 08:20:13.801959 12340 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786177179845590-12340-0/krb5kdc/krb5cc' of type FILE
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.802963 12340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:13.803817 12340 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[20729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:20:13.812141 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:13.804597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.12.13.1:35431)
0808 08:20:13.804708 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:13.804712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:13.804850 (+ 138us) server_negotiation.cc:407] Connection header received
0808 08:20:13.804928 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:13.804930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:13.804966 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:13.805030 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:13.805459 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.806038 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.806741 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:13.806953 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:13.807031 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:13.809256 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:13.809259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:13.809261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:13.809284 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:13.810631 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.811032 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.811035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.811036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.811112 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:13.811377 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:13.811380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:13.811381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:13.811539 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:13.811641 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:13.811836 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 08:20:13.811958 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:20:13.813226 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 20882
I20260808 08:20:13.827361 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 21095
I20260808 08:20:13.850507 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 21290
I20260808 08:20:13.869357 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 20743
2026-08-08T08:20:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2489 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[21506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[21506](info): set up 2 sockets
Aug 08 08:20:13 dist-test-slave-6ntk krb5kdc[21506](info): commencing operation
krb5kdc: starting...
W20260808 08:20:15.163465 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:15 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab.
Entry for principal oryx/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab.
I20260808 08:20:15.302090 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33243
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab
--principal=oryx/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:15.418923 21525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:15.419238 21525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:15.419343 21525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:15.423378 21525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:15.423463 21525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:15.423501 21525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:15.423558 21525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:15.423595 21525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:15.429144 21525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab
--principal=oryx/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:15.430347 21525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:15.431432 21525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:15.440423 21534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:15.440464 21525 server_base.cc:1034] running on GCE node
W20260808 08:20:15.440488 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:15.440433 21535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:15.441262 21525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:15.442745 21525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:15.443970 21525 hybrid_clock.cc:648] HybridClock initialized: now 1786177215443935 us; error 48 us; skew 500 ppm
Aug 08 08:20:15 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:15.447043 21525 init.cc:377] Logged in from keytab as oryx/127.12.13.62@KRBTEST.COM (short username oryx)
I20260808 08:20:15.447942 21525 webserver.cc:460] Webserver started at http://127.12.13.62:42869/ using document root <none> and password file <none>
I20260808 08:20:15.448503 21525 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:15.448565 21525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:15.448807 21525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:15.451520 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "a78fd6a525bd4b3e942061d1d81d610e"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "0be32e4098ecd567270c4f77b2b0b826"
server_key_iv: "ef785bf5b143719cf5d1aeaac64299f9"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.452193 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "a78fd6a525bd4b3e942061d1d81d610e"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "0be32e4098ecd567270c4f77b2b0b826"
server_key_iv: "ef785bf5b143719cf5d1aeaac64299f9"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.456379 21525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:15.459190 21547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:15.460538 21525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:20:15.460700 21525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "a78fd6a525bd4b3e942061d1d81d610e"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "0be32e4098ecd567270c4f77b2b0b826"
server_key_iv: "ef785bf5b143719cf5d1aeaac64299f9"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.460829 21525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:15.478960 21525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:15.482566 21525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:15.482787 21525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:15.489826 21525 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:33243
I20260808 08:20:15.490231 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:33243 every 8 connection(s)
I20260808 08:20:15.490950 21525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:15.493783 21631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:15.499382 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Bootstrap starting.
I20260808 08:20:15.499619 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 21525
I20260808 08:20:15.499769 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:15.500084 12340 external_mini_cluster.cc:1428] Setting key 21c9046ab2c6ff4d0d26655d989a920c
I20260808 08:20:15.502244 21631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:15.503022 21631 log.cc:824] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:15.505070 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: No bootstrap required, opened a new log
I20260808 08:20:15.508644 21631 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } }
I20260808 08:20:15.508900 21631 raft_consensus.cc:374] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:15.508982 21631 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78fd6a525bd4b3e942061d1d81d610e, State: Initialized, Role: FOLLOWER
Aug 08 08:20:15 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:15.509521 21631 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } }
I20260808 08:20:15.509690 21631 raft_consensus.cc:388] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:15.509764 21631 raft_consensus.cc:482] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:15.509888 21631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:15.510852 21631 raft_consensus.cc:504] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } }
I20260808 08:20:15.511209 21631 leader_election.cc:302] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78fd6a525bd4b3e942061d1d81d610e; no voters:
I20260808 08:20:15.511582 21631 leader_election.cc:288] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:15.511675 21640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:15.511909 21640 raft_consensus.cc:686] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 LEADER]: Becoming Leader. State: Replica: a78fd6a525bd4b3e942061d1d81d610e, State: Running, Role: LEADER
I20260808 08:20:15.512269 21640 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } }
I20260808 08:20:15.512959 21631 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:15.513548 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78fd6a525bd4b3e942061d1d81d610e. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } }
I20260808 08:20:15.514369 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:15.514230 21641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } }
I20260808 08:20:15.514681 21641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:15.516494 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:15.501540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.12.13.62:33243)
0808 08:20:15.502095 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:15.502111 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:15.502150 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:20:15.502890 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:15.502910 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:15.503316 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:15.503681 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:15.504567 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:15.505653 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:15.506608 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:15.507096 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:15.507278 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:15.510174 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:15.510178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:15.510192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:15.510222 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:15.512291 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:15.512770 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:15.512783 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:15.512788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:15.512956 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:15.513254 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:15.513257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:15.513259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:15.513815 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:15.514842 (+ 1027us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:15.515019 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 08:20:15.515383 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":168,"threads_started":1}
W20260808 08:20:15.518322 21669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:20:15.518400 21669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:20:15.518460 21656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:15.519328 21656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:15.526254 21656 catalog_manager.cc:1347] Generated new cluster ID: dd7d30563e8340cbb42cf8185b0a59cb
I20260808 08:20:15.526337 21656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:15.542194 21656 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:15.543584 21656 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:15.551573 21656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Generated new TSK 0
I20260808 08:20:15.552459 21656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab.
Entry for principal oryx/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab.
I20260808 08:20:15.591634 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33243
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:15.713805 21676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:15.714130 21676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:15.714214 21676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:15.718559 21676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:15.718617 21676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:15.718705 21676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:15.724006 21676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:15.725126 21676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:15.726385 21676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:15.736675 21687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:15.736698 21689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:15.736675 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:15.736953 21676 server_base.cc:1034] running on GCE node
I20260808 08:20:15.737563 21676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:15.738303 21676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:15.739472 21676 hybrid_clock.cc:648] HybridClock initialized: now 1786177215739452 us; error 32 us; skew 500 ppm
Aug 08 08:20:15 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:15.742669 21676 init.cc:377] Logged in from keytab as oryx/127.12.13.1@KRBTEST.COM (short username oryx)
I20260808 08:20:15.743750 21676 webserver.cc:460] Webserver started at http://127.12.13.1:44709/ using document root <none> and password file <none>
I20260808 08:20:15.744354 21676 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:15.744416 21676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:15.744674 21676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:15.746858 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "307ad2ac0a53478a8deda7ad2f09a713"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "2e79ecfd245ceddad60c482e88c814a9"
server_key_iv: "203264c837edc4fb9bbe9437b7920add"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.747427 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "307ad2ac0a53478a8deda7ad2f09a713"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "2e79ecfd245ceddad60c482e88c814a9"
server_key_iv: "203264c837edc4fb9bbe9437b7920add"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.751511 21676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:15.754172 21707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:15.755604 21676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:20:15.755775 21676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "307ad2ac0a53478a8deda7ad2f09a713"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "2e79ecfd245ceddad60c482e88c814a9"
server_key_iv: "203264c837edc4fb9bbe9437b7920add"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.755904 21676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:15.770680 21676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:15.774145 21676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:15.774331 21676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:15.774923 21676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:15.775818 21676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:15.775877 21676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:15.775928 21676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:15.775949 21676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:15.786466 21676 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:38235
I20260808 08:20:15.786525 21886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:38235 every 8 connection(s)
I20260808 08:20:15.787539 21676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:15.789897 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 21676
I20260808 08:20:15.790100 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:15.790664 12340 external_mini_cluster.cc:1428] Setting key 0453c6d70e76c7f0fc266204a2e23e83
Aug 08 08:20:15 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:15.804563 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:15.789609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:60217 (local address 127.12.13.62:33243)
0808 08:20:15.789809 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:15.789816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:15.790990 (+ 1174us) server_negotiation.cc:407] Connection header received
0808 08:20:15.792684 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:15.792686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:15.792730 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:15.792788 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:15.794865 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:15.795759 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:15.797116 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:15.797561 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:15.797691 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:15.800619 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:15.800623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:15.800626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:15.800659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:15.802426 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:15.803033 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:15.803037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:15.803038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:15.803129 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:15.803442 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:15.803445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:15.803446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:15.803620 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:15.803695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:15.804305 (+ 610us) server_negotiation.cc:299] Negotiation successful
0808 08:20:15.804431 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 08:20:15.805534 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:15.790093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.1:60217)
0808 08:20:15.790775 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:15.790849 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:15.792425 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:15.793006 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:15.793020 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:15.793721 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:15.794583 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:15.794610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:15.795918 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:15.795921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:15.796970 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:15.796980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:15.797745 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:15.797749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:15.797994 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:15.798891 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:15.798909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:15.800416 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:15.802557 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:15.802562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:15.802576 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:15.802908 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:15.803211 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:15.803214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:15.803215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:15.803352 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:15.803707 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:15.803712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:15.804043 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 08:20:15.804238 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 08:20:15.804524 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.2@KRBTEST.COM" created.
I20260808 08:20:15.806972 21889 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33243
I20260808 08:20:15.807231 21889 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:15.807757 21889 heartbeater.cc:511] Master 127.12.13.62:33243 requested a full tablet report, sending...
I20260808 08:20:15.809303 21569 ts_manager.cc:194] Registered new tserver with Master: 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235)
I20260808 08:20:15.811002 21569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:60217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab.
Entry for principal oryx/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab.
I20260808 08:20:15.825492 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33243
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:15.942094 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:15.942363 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:15.942408 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:15.946421 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:15.946476 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:15.946535 21896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:15.951843 21896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:15.952970 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:15.954015 21896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:15.964643 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:15.964625 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:15.964706 21904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:15.964789 21896 server_base.cc:1034] running on GCE node
I20260808 08:20:15.965227 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:15.966092 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:15.967289 21896 hybrid_clock.cc:648] HybridClock initialized: now 1786177215967278 us; error 37 us; skew 500 ppm
Aug 08 08:20:15 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:15.970136 21896 init.cc:377] Logged in from keytab as oryx/127.12.13.2@KRBTEST.COM (short username oryx)
I20260808 08:20:15.971066 21896 webserver.cc:460] Webserver started at http://127.12.13.2:43747/ using document root <none> and password file <none>
I20260808 08:20:15.971683 21896 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:15.971776 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:15.972028 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:15.974246 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "948827c379784a05be72ff83b08d296d"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "87639643bc2a4bb5f3a786c86c81a50d"
server_key_iv: "ae43e7895886a70a34b869dfaf7ddde6"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.974821 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "948827c379784a05be72ff83b08d296d"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "87639643bc2a4bb5f3a786c86c81a50d"
server_key_iv: "ae43e7895886a70a34b869dfaf7ddde6"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.978780 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:15.981488 21917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:15.983003 21896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:20:15.983191 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "948827c379784a05be72ff83b08d296d"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "87639643bc2a4bb5f3a786c86c81a50d"
server_key_iv: "ae43e7895886a70a34b869dfaf7ddde6"
server_key_version: "encryptionkey@0"
I20260808 08:20:15.983321 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:16.037824 21896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:16.041225 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:16.041464 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:16.042188 21896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:16.043154 21896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:16.043237 21896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.043323 21896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:16.043382 21896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.054239 21896 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:41789
I20260808 08:20:16.054265 22087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:41789 every 8 connection(s)
I20260808 08:20:16.055323 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:16.064106 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 21896
I20260808 08:20:16.064280 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:16.064623 12340 external_mini_cluster.cc:1428] Setting key ad49bc699600619fd98dace246ab8f27
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:16.071312 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.057414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:36705 (local address 127.12.13.62:33243)
0808 08:20:16.057642 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.057647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.058638 (+ 991us) server_negotiation.cc:407] Connection header received
0808 08:20:16.060071 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.060074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.060126 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.060204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.061544 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.062137 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.063352 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.063563 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.063643 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.066883 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.066890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.066895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.066932 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.068797 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.069566 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.069568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.069570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.069647 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.070105 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.070108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.070109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.070293 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.070378 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.071078 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.071186 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 08:20:16.072757 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.057843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:36705)
0808 08:20:16.058467 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.058516 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.059796 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.060430 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.060438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.060828 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.061409 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.061423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.062299 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.062301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.063221 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.063231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.063715 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.063717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.063878 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.064634 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.064650 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.066680 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.068946 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.068956 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.068977 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.069445 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.069745 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.069750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.069754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.070002 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.070409 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.070419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.070797 (+ 378us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.071025 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.071440 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260808 08:20:16.074141 22088 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33243
I20260808 08:20:16.074448 22088 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:16.075111 22088 heartbeater.cc:511] Master 127.12.13.62:33243 requested a full tablet report, sending...
I20260808 08:20:16.076402 21569 ts_manager.cc:194] Registered new tserver with Master: 948827c379784a05be72ff83b08d296d (127.12.13.2:41789)
WARNING: no policy specified for oryx/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:20:16.077064 21569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:36705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab.
Entry for principal oryx/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab.
I20260808 08:20:16.097533 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33243
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:16.222626 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:16.222966 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:16.223050 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:16.227485 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:16.227584 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:16.227679 22096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:16.233284 22096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:16.234681 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:16.235809 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:16.246474 22109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:16.246467 22108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:16.246520 22096 server_base.cc:1034] running on GCE node
W20260808 08:20:16.246467 22113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:16.247238 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:16.247954 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:16.249133 22096 hybrid_clock.cc:648] HybridClock initialized: now 1786177216249078 us; error 74 us; skew 500 ppm
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:16.252305 22096 init.cc:377] Logged in from keytab as oryx/127.12.13.3@KRBTEST.COM (short username oryx)
I20260808 08:20:16.253437 22096 webserver.cc:460] Webserver started at http://127.12.13.3:34869/ using document root <none> and password file <none>
I20260808 08:20:16.254204 22096 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:16.254276 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:16.254475 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:16.256698 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "930622da90e04994b43dc0f72fafdcba"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "ebdd7ecb7907808a770432ffd72b277c"
server_key_iv: "a139876625c2b91c9de78d441848f19b"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.257251 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "930622da90e04994b43dc0f72fafdcba"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "ebdd7ecb7907808a770432ffd72b277c"
server_key_iv: "a139876625c2b91c9de78d441848f19b"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.261576 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 08:20:16.264240 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.265728 22096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:16.265867 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "930622da90e04994b43dc0f72fafdcba"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "ebdd7ecb7907808a770432ffd72b277c"
server_key_iv: "a139876625c2b91c9de78d441848f19b"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.266047 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:16.277062 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:16.280238 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:16.280472 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:16.281128 22096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:16.282126 22096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:16.282177 22096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.282250 22096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:16.282294 22096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.292317 22096 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:44389
I20260808 08:20:16.292339 22293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:44389 every 8 connection(s)
I20260808 08:20:16.293284 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:16.295636 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 22096
I20260808 08:20:16.295799 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:16.296099 12340 external_mini_cluster.cc:1428] Setting key c1f754e1532daaa05d2e18d5fd010d56
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:16.307308 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.295245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:56655 (local address 127.12.13.62:33243)
0808 08:20:16.295491 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.295498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.296491 (+ 993us) server_negotiation.cc:407] Connection header received
0808 08:20:16.298122 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.298125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.298170 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.298255 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.299539 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.300067 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.301161 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.301382 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.301459 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.303888 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.303891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.303893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.303918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.305181 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.305668 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.305671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.305672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.305747 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.306149 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.306154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.306156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.306403 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.306519 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.307005 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.307132 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:20:16.308151 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.295707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:56655)
0808 08:20:16.296286 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.296366 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.297850 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.298469 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.298480 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.298893 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.299406 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.299420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.300177 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.300180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.301055 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.301064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.301512 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.301515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.301662 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.302219 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.302235 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.303710 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.305288 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.305293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.305304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.305580 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.305827 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.305829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.305830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.306048 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.306514 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.306521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.306762 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.306945 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.307267 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":164,"threads_started":1}
I20260808 08:20:16.309545 22294 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33243
I20260808 08:20:16.309816 22294 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:16.310475 22294 heartbeater.cc:511] Master 127.12.13.62:33243 requested a full tablet report, sending...
I20260808 08:20:16.311472 21569 ts_manager.cc:194] Registered new tserver with Master: 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:16.312054 21569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:56655
I20260808 08:20:16.322278 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.61.keytab.
Entry for principal oryx/127.12.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.61.keytab.
I20260808 08:20:16.353078 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:46093
--webserver_interface=127.12.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.61.keytab
--principal=oryx/127.12.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33243,127.12.13.61:46093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:16.475473 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:16.475785 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:16.475827 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:16.479933 22304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:16.480001 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:16.480021 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:16.480038 22304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:16.480053 22304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:16.485513 22304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33243,127.12.13.61:46093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.61.keytab
--principal=oryx/127.12.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:46093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
W20260808 08:20:16.485786 22304 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.13.62:33243,127.12.13.61:46093'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 08:20:16.486891 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:16.488046 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:16.496964 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:16.496974 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:16.497207 22304 server_base.cc:1034] running on GCE node
W20260808 08:20:16.497493 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:16.498203 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:16.499774 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:16.500958 22304 hybrid_clock.cc:648] HybridClock initialized: now 1786177216500890 us; error 86 us; skew 500 ppm
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:16.504030 22304 init.cc:377] Logged in from keytab as oryx/127.12.13.61@KRBTEST.COM (short username oryx)
I20260808 08:20:16.505012 22304 webserver.cc:460] Webserver started at http://127.12.13.61:35887/ using document root <none> and password file <none>
I20260808 08:20:16.505600 22304 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:16.505666 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:16.506016 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:16.508136 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/instance:
uuid: "5ff3ab74c44b4b3398655f227749a56e"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "eeb5cadc567c82c7aaa8ce6795da4437"
server_key_iv: "76a4e9c185f23594f20276768237cd59"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.508725 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal/instance:
uuid: "5ff3ab74c44b4b3398655f227749a56e"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "eeb5cadc567c82c7aaa8ce6795da4437"
server_key_iv: "76a4e9c185f23594f20276768237cd59"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.512741 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:20:16.515594 22327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.516883 22304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:16.517019 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
uuid: "5ff3ab74c44b4b3398655f227749a56e"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "eeb5cadc567c82c7aaa8ce6795da4437"
server_key_iv: "76a4e9c185f23594f20276768237cd59"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.517129 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:16.529256 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:16.532356 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:16.532572 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:16.540253 22304 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.61:46093
I20260808 08:20:16.540555 22397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.61:46093 every 8 connection(s)
I20260808 08:20:16.541371 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
I20260808 08:20:16.544371 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:16.548280 22399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:20:16.551090 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 22304
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.61@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
I20260808 08:20:16.563071 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.550342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.12.13.62:33243)
0808 08:20:16.550487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.550490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.551322 (+ 832us) server_negotiation.cc:407] Connection header received
0808 08:20:16.552631 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.552634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.552706 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.552817 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.553805 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.554439 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.555503 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.555727 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.555840 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.558808 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.558812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.558818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.558870 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.560739 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.561433 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.561437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.561439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.561534 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.561926 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.561929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.561931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.562177 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.562293 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.562754 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.562892 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:20:16.563998 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.552095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.12.13.61:46093)
0808 08:20:16.552454 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.552467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.552512 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:20:16.552674 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.552678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.552872 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.553058 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.553934 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.554807 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.555653 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.556040 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.556159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.559266 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.559289 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.559308 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.559336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.561233 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.561707 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.561713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.561715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.561801 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.562130 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.562133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.562135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.562507 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.562721 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.562949 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.563094 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"spinlock_wait_cycles":9472,"thread_start_us":121,"threads_started":1}
I20260808 08:20:16.564044 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.550623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41230)
0808 08:20:16.551156 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.551211 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.552391 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.552882 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.552892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.553269 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.553649 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.553663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.554580 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.554583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.555363 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.555373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.555899 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.555903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.556078 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.556737 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.556756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.558598 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.560882 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.560890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.560909 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.561329 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.561641 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.561647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.561649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.561798 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.562311 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.562319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.562526 (+ 207us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.562707 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.563035 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"spinlock_wait_cycles":896,"thread_start_us":176,"threads_started":1}
I20260808 08:20:16.565443 22399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.61@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
I20260808 08:20:16.573195 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.566008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56178)
0808 08:20:16.566133 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.566147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.566227 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.566474 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.566477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.566603 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.566689 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.566694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.567401 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.567404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.567897 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.567904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.568298 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.568300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.568395 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.568766 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.568777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.570043 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.572023 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.572025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.572026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.572206 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.572443 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.572445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.572446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.572508 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.572817 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.572819 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.572893 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.572971 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.573062 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 08:20:16.573565 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.566067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.12.13.61:46093)
0808 08:20:16.566163 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.566169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.566187 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:20:16.566318 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.566321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.566374 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.566444 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.566786 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.567275 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.567990 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.568208 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.568281 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.570160 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.570163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.570165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.570186 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.571926 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.572290 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.572292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.572293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.572382 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.572580 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.572583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.572584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.572737 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.572811 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.573029 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.573115 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 08:20:16.576773 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Bootstrap starting.
I20260808 08:20:16.579247 22399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:16.579957 22399 log.cc:824] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:16.582079 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: No bootstrap required, opened a new log
I20260808 08:20:16.585119 22399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } }
I20260808 08:20:16.585331 22399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:16.585366 22399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff3ab74c44b4b3398655f227749a56e, State: Initialized, Role: FOLLOWER
I20260808 08:20:16.585757 22399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } }
I20260808 08:20:16.586306 22411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } } }
I20260808 08:20:16.586459 22411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:16.587144 22399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: configured and running, proceeding with master startup.
W20260808 08:20:16.592124 22431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:20:16.592221 22431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:20:16.594954 22304 master_runner.cc:428] Detected that this master 5ff3ab74c44b4b3398655f227749a56e is joining an existing cluster
I20260808 08:20:16.595007 22304 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.13.61:46093
I20260808 08:20:16.605695 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.597637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.12.13.61:46093)
0808 08:20:16.597754 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.597758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.597772 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:16.597811 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.597814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.597858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.597972 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.598577 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.599365 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.600415 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.600634 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.600745 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.602011 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.602014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.602017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.602040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.603806 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.604329 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.604335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.604338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.604442 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.604770 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.604774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.604777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.605009 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.605099 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.605391 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.605530 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":42}
I20260808 08:20:16.605708 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.597203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56186)
0808 08:20:16.597505 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.597523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.597622 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.597987 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.597993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.598276 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.598435 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.598445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.599482 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.599487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.600314 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.600322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.600763 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.600766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.600872 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.601421 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.601441 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.601834 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.603925 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.603928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.603930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.604234 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.604548 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.604551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.604554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.604654 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.605140 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.605142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.605210 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.605321 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.605474 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260808 08:20:16.607534 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.597185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.12.13.62:33243)
0808 08:20:16.597319 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.597325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.597684 (+ 359us) server_negotiation.cc:407] Connection header received
0808 08:20:16.597849 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.597853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.597944 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.598021 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.598607 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.599143 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.600363 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.600735 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.600846 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.603101 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.603110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.603113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.603147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.604961 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.605456 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.605460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.605462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.605560 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.606569 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.606573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.606574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.606799 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.606908 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.607231 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.607364 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 08:20:16.608014 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.597288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41232)
0808 08:20:16.597602 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.597613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.597746 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.598149 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.598151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.598356 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.598462 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.598470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.599259 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.599264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.600219 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.600229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.600977 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.600981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.601083 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.602161 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.602178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.602668 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.605070 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.605073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.605075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.605328 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.605675 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.605677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.605679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.606462 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.606912 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.606915 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.607022 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.607689 (+ 667us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.607842 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.62@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
I20260808 08:20:16.622856 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.612540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56188)
0808 08:20:16.613040 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.613084 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.613241 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.613648 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.613654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.614108 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.614220 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.614226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.615096 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.615100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.615748 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.615757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.616227 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.616230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.616353 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.616951 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.616967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.618936 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.621044 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.621055 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.621063 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.621393 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.621705 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.621707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.621709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.621797 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.622289 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.622295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.622420 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.622549 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.622713 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":113,"threads_started":1}
I20260808 08:20:16.622905 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.612729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.12.13.61:46093)
0808 08:20:16.612849 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.612854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.613163 (+ 309us) server_negotiation.cc:407] Connection header received
0808 08:20:16.613389 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.613391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.613524 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.613611 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.614335 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.614961 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.615891 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.616112 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.616222 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.619090 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.619094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.619096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.619127 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.620908 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.621524 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.621527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.621529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.621622 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.621953 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.621956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.621958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.622173 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.622281 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.622633 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.622762 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 08:20:16.624615 21569 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.13.61:46093: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e"
member_type: NON_VOTER
last_known_addr {
host: "127.12.13.61"
port: 46093
}
attrs {
promote: true
}
}
dest_uuid: "a78fd6a525bd4b3e942061d1d81d610e"
cas_config_opid_index: -1
I20260808 08:20:16.625281 21569 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } }
I20260808 08:20:16.627379 22449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5ff3ab74c44b4b3398655f227749a56e (127.12.13.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } }
I20260808 08:20:16.628037 22363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.12.13.60@KRBTEST.COM; defaulting to no policy
I20260808 08:20:16.629459 22449 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.13.61:46093
I20260808 08:20:16.629859 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } }
I20260808 08:20:16.630012 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:16.630034 22304 master.cc:537] Master@127.12.13.61:46093 shutting down...
I20260808 08:20:16.630142 22363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Refusing update from remote peer a78fd6a525bd4b3e942061d1d81d610e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 08:20:16.630923 22450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.60@KRBTEST.COM" created.
W20260808 08:20:16.634038 21551 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 08:20:16.634979 21551 consensus_peers.cc:487] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e -> Peer 5ff3ab74c44b4b3398655f227749a56e (127.12.13.61:46093): Couldn't send request to peer 5ff3ab74c44b4b3398655f227749a56e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 08:20:16.635665 22304 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:16.635963 22304 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:16.636075 22304 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: stopping tablet replica
I20260808 08:20:16.642009 22304 master.cc:552] Master@127.12.13.61:46093 shutdown complete.
I20260808 08:20:16.642068 22304 master_runner.cc:305] Clearing existing system tablet
I20260808 08:20:16.642340 22304 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:16.642401 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:16.643730 22304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:16.645574 22457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.645792 22304 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:16.646082 22304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:16.646124 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
uuid: "5ff3ab74c44b4b3398655f227749a56e"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "eeb5cadc567c82c7aaa8ce6795da4437"
server_key_iv: "76a4e9c185f23594f20276768237cd59"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.648600 22304 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:16.651451 22304 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 08:20:16.651530 22304 log.cc:1196] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 08:20:16.652608 22304 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Deleting consensus metadata
I20260808 08:20:16.652801 22304 master_runner.cc:315] Copying system tablet from 127.12.13.62:33243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.60.keytab.
Entry for principal oryx/127.12.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.60.keytab.
I20260808 08:20:16.654300 22304 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: tablet copy: Beginning tablet copy session from remote peer at address 127.12.13.62:33243
I20260808 08:20:16.654692 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:36387
--webserver_interface=127.12.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.60.keytab
--principal=oryx/127.12.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:20:16.663612 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.655064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.12.13.62:33243)
0808 08:20:16.655227 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.655232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.655641 (+ 409us) server_negotiation.cc:407] Connection header received
0808 08:20:16.655778 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.655781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.655837 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.655934 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.656405 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.657310 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.658537 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.658774 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.658889 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.660242 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.660249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.660252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.660279 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.661925 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.662373 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.662377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.662379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.662462 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.662742 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.662746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.662747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.662940 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.663050 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.663320 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.663449 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:20:16.663621 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.655244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41244)
0808 08:20:16.655493 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.655507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.655678 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.655951 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.655955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.656162 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.656277 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.656283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.657461 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.657467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.658412 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.658422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.658897 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.658901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.658995 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.659705 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.659718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.660067 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.662040 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.662046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.662048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.662263 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.662557 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.662561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.662563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.662651 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.663037 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.663040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.663150 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.663259 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.663412 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260808 08:20:16.664538 21604 tablet_copy_service.cc:140] P a78fd6a525bd4b3e942061d1d81d610e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5ff3ab74c44b4b3398655f227749a56e ({username='oryx', principal='oryx/127.12.13.61@KRBTEST.COM'} at 127.0.0.1:41244)
I20260808 08:20:16.664736 21604 tablet_copy_service.cc:161] P a78fd6a525bd4b3e942061d1d81d610e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5ff3ab74c44b4b3398655f227749a56e at {username='oryx', principal='oryx/127.12.13.61@KRBTEST.COM'} at 127.0.0.1:41244: session id = 5ff3ab74c44b4b3398655f227749a56e-00000000000000000000000000000000
I20260808 08:20:16.666457 21604 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 08:20:16.668000 22304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 08:20:16.670127 22304 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: tablet copy: Starting download of 0 data blocks...
I20260808 08:20:16.670349 22304 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: tablet copy: Starting download of 1 WAL segments...
I20260808 08:20:16.672166 22304 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 08:20:16.673837 21604 tablet_copy_service.cc:342] P a78fd6a525bd4b3e942061d1d81d610e: Request end of tablet copy session 5ff3ab74c44b4b3398655f227749a56e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.13.61@KRBTEST.COM'} at 127.0.0.1:41244
I20260808 08:20:16.674044 21604 tablet_copy_service.cc:433] P a78fd6a525bd4b3e942061d1d81d610e: ending tablet copy session 5ff3ab74c44b4b3398655f227749a56e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5ff3ab74c44b4b3398655f227749a56e
I20260808 08:20:16.681566 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:16.683447 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:16.683609 22473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:16.683897 22474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:16.687037 22304 server_base.cc:1034] running on GCE node
I20260808 08:20:16.687242 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:16.687613 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:16.688773 22304 hybrid_clock.cc:648] HybridClock initialized: now 1786177216688753 us; error 38 us; skew 500 ppm
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:16.690405 22304 init.cc:377] Logged in from keytab as oryx/127.12.13.61@KRBTEST.COM (short username oryx)
I20260808 08:20:16.690900 22304 webserver.cc:460] Webserver started at http://127.12.13.61:44113/ using document root <none> and password file <none>
I20260808 08:20:16.691107 22304 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:16.691162 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:16.692351 22304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:16.693663 22485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.693923 22304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:16.694021 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
uuid: "5ff3ab74c44b4b3398655f227749a56e"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "eeb5cadc567c82c7aaa8ce6795da4437"
server_key_iv: "76a4e9c185f23594f20276768237cd59"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.694129 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:16.701220 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:16.703306 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:16.708395 22304 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.61:46093
I20260808 08:20:16.708966 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
I20260808 08:20:16.709386 22564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.61:46093 every 8 connection(s)
I20260808 08:20:16.710764 22566 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:20:16.711589 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Bootstrap starting.
I20260808 08:20:16.720132 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:20:16.720475 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Bootstrap complete.
I20260808 08:20:16.720999 22566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } }
I20260808 08:20:16.721136 22566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ff3ab74c44b4b3398655f227749a56e, State: Initialized, Role: LEARNER
I20260808 08:20:16.721253 22566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } }
I20260808 08:20:16.721628 22570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } }
I20260808 08:20:16.721732 22566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:16.721804 22570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: LEARNER
I20260808 08:20:16.724669 22587 catalog_manager.cc:1259] Loaded cluster ID: dd7d30563e8340cbb42cf8185b0a59cb
I20260808 08:20:16.724786 22587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: loading cluster ID for follower catalog manager: success
I20260808 08:20:16.726516 22587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: acquiring CA information for follower catalog manager: success
I20260808 08:20:16.727595 22587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:20:16.813728 21889 heartbeater.cc:503] Master 127.12.13.62:33243 was elected leader, sending a full tablet report...
W20260808 08:20:16.821950 22471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:16.822250 22471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:16.822326 22471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:16.826730 22471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:16.826794 22471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:16.826814 22471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:16.826831 22471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:16.826848 22471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:16.832564 22471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.60.keytab
--principal=oryx/127.12.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:16.833715 22471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:16.834872 22471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:16.843726 22600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:16.843746 22594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:16.843804 22595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:16.844146 22471 server_base.cc:1034] running on GCE node
I20260808 08:20:16.844727 22471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:16.845927 22471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:16.847137 22471 hybrid_clock.cc:648] HybridClock initialized: now 1786177216847123 us; error 36 us; skew 500 ppm
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:16.850142 22471 init.cc:377] Logged in from keytab as oryx/127.12.13.60@KRBTEST.COM (short username oryx)
I20260808 08:20:16.851071 22471 webserver.cc:460] Webserver started at http://127.12.13.60:45507/ using document root <none> and password file <none>
I20260808 08:20:16.851684 22471 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:16.851753 22471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:16.851980 22471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:16.854477 22471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/instance:
uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "11d535b959ffd58f1f4491d85b560755"
server_key_iv: "6d03ae6d9e196c9cda225618534a2342"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.855163 22471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal/instance:
uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "11d535b959ffd58f1f4491d85b560755"
server_key_iv: "6d03ae6d9e196c9cda225618534a2342"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.859580 22471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:16.862460 22612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:16.863860 22471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:16.863991 22471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "11d535b959ffd58f1f4491d85b560755"
server_key_iv: "6d03ae6d9e196c9cda225618534a2342"
server_key_version: "encryptionkey@0"
I20260808 08:20:16.864099 22471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:16.879232 22471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:16.882364 22471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:16.882570 22471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:16.890498 22471 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.60:36387
I20260808 08:20:16.890803 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.60:36387 every 8 connection(s)
I20260808 08:20:16.891640 22471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
I20260808 08:20:16.894766 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:16.897015 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 22471
I20260808 08:20:16.899286 22693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.13.60@KRBTEST.COM
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.60@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:16.912429 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.901695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.12.13.62:33243)
0808 08:20:16.901836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.901840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.901978 (+ 138us) server_negotiation.cc:407] Connection header received
0808 08:20:16.902123 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.902126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.902181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.902272 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.902972 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.903492 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.904885 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.905257 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.905350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.908705 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.908726 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.908734 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.908767 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.910421 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.911067 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.911071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.911072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.911175 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.911528 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.911534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.911538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.911771 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.911851 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.912195 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.912288 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 08:20:16.912932 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.901428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41248)
0808 08:20:16.901818 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.901891 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.902024 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.902332 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.902336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.902661 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.902831 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.902840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.903628 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.903631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.904732 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.904743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.905563 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.905569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.905701 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.906216 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.906243 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.908550 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.910707 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.910714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.910719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.910959 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.911284 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.911288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.911289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.911416 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.911879 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.911886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.911998 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.912145 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.912313 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"spinlock_wait_cycles":7424,"thread_start_us":128,"threads_started":1}
I20260808 08:20:16.912935 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.898690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.12.13.60:36387)
0808 08:20:16.899174 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.899189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.899225 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:20:16.900002 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.900021 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.900407 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.900738 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.901606 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.902887 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.903813 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.904542 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.904733 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.907860 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.907866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.907875 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.907896 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.909760 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.910272 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.910276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.910280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.910416 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.910716 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.910718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.910719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.911102 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.911298 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.911593 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.911909 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":345,"thread_start_us":142,"threads_started":1}
I20260808 08:20:16.914542 22693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.60@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
I20260808 08:20:16.922418 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.915079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56196)
0808 08:20:16.915234 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.915247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.915320 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.915743 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.915746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.915878 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.915965 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.915970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.916826 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.916828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.917349 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.917358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.917763 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.917765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.917834 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.918280 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.918290 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.919435 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.921152 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.921154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.921155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.921327 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.921568 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.921570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.921571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.921642 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.922045 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.922047 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.922121 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.922205 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.922299 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 08:20:16.922490 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.915206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.12.13.61:46093)
0808 08:20:16.915546 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.915549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.915562 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:20:16.915597 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.915599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.915637 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.915713 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.916067 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.916734 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.917453 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.917666 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.917749 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.919563 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.919566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.919568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.919592 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.921060 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.921410 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.921413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.921414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.921490 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.921718 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.921721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.921722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.921959 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.922040 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.922266 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.922368 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":99,"threads_started":1}
I20260808 08:20:16.923290 22693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.60@KRBTEST.COM for oryx/127.12.13.60@KRBTEST.COM
I20260808 08:20:16.930647 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.923732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47602)
0808 08:20:16.923875 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.923887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.923947 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.924168 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.924170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.924291 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.924359 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.924365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.925058 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.925060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.925535 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.925542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.926054 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.926057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.926111 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.926483 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.926493 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.927616 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.929265 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.929268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.929270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.929515 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.929772 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.929774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.929775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.929852 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.930231 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.930234 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.930312 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.930402 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.930507 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 08:20:16.930711 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.923889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.12.13.60:36387)
0808 08:20:16.923996 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.924000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.924019 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:20:16.924046 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.924048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.924079 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.924143 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.924453 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.924957 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.925632 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.925949 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.926044 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.927740 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.927743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.927745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.927766 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.929149 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.929607 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.929609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.929610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.929686 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.929968 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.929970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.929971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.930129 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.930205 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.930450 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.930564 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":38}
I20260808 08:20:16.934120 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Bootstrap starting.
I20260808 08:20:16.936617 22693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:16.937364 22693 log.cc:824] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:16.939399 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: No bootstrap required, opened a new log
I20260808 08:20:16.942497 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } }
I20260808 08:20:16.942706 22693 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:16.942791 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e9d8a05b014a7691b9c07e4fd32053, State: Initialized, Role: FOLLOWER
I20260808 08:20:16.943257 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } }
I20260808 08:20:16.943779 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } } }
I20260808 08:20:16.943938 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:16.944514 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 08:20:16.949414 22724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:20:16.949505 22724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:20:16.952821 22471 master_runner.cc:428] Detected that this master d6e9d8a05b014a7691b9c07e4fd32053 is joining an existing cluster
I20260808 08:20:16.952896 22471 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.13.60:36387
I20260808 08:20:16.965179 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.955928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.12.13.60:36387)
0808 08:20:16.956033 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.956038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.956375 (+ 337us) server_negotiation.cc:407] Connection header received
0808 08:20:16.956523 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.956526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.956560 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.956604 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.957246 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.958039 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959359 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.959711 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959784 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.961330 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.961334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.961336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.961358 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.963219 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.963780 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.963786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.963790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.963873 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.964235 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.964237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.964239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.964423 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.964503 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.964805 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.964969 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":45}
I20260808 08:20:16.965199 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.955245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.12.13.62:33243)
0808 08:20:16.955386 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.955391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.955656 (+ 265us) server_negotiation.cc:407] Connection header received
0808 08:20:16.955818 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.955821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.955881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.955956 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.956646 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.957455 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.958732 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.959084 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959159 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.961532 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.961535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.961538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.961574 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.963210 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.963728 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.963732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.963735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.963836 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.964160 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.964167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.964171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.964440 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.964535 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.964910 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.965036 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:20:16.965468 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.955946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47618)
0808 08:20:16.956293 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.956308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.956434 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.956728 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.956731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.957008 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.957128 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.957136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.958181 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.958184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.959230 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.959241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.959986 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.960086 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.960581 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.960596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.961199 (+ 603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.963333 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.963336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.963338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.963691 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.964020 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.964022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.964023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.964139 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.964567 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.964569 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.964650 (+ 81us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.964758 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.964914 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":158,"spinlock_wait_cycles":3456,"thread_start_us":144,"threads_started":1}
I20260808 08:20:16.965755 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.955212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41258)
0808 08:20:16.955556 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.955568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.955702 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.956085 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.956088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.956355 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.956516 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.956527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.957627 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.957630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.958599 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.958608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.959337 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.959421 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.960073 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.960089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.961386 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.963327 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.963334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.963336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.963624 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.963939 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.963942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.963944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.964038 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.964536 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.964539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.964641 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.965435 (+ 794us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.965577 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":3,"spinlock_wait_cycles":2304,"thread_start_us":99,"threads_started":1}
I20260808 08:20:16.967008 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 21676
Aug 08 08:20:16 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.62@KRBTEST.COM for oryx/127.12.13.60@KRBTEST.COM
I20260808 08:20:16.985509 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:38235
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:33243
--webserver_port=44709
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:20:16.987516 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.955360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.12.13.61:46093)
0808 08:20:16.955480 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.955484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.956362 (+ 878us) server_negotiation.cc:407] Connection header received
0808 08:20:16.956559 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.956565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.956622 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.956675 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.957301 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.958109 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959330 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.959726 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.959815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.983081 (+ 23266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.983086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.983090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.983126 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.984942 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.985465 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.985470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.985471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.985563 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.985944 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.985948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.985951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.986235 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.986344 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.987121 (+ 777us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.987274 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 08:20:16.988226 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.955213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56210)
0808 08:20:16.956265 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.956280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.956441 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.956811 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.956814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.957043 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.957167 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.957173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.958242 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.958245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.959202 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.959211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.959999 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.960005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.960132 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.982348 (+ 22216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.982365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.982873 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.985080 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.985084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.985086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.985338 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.985677 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.985682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.985686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.985766 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.986713 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.986715 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.986902 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.987760 (+ 858us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.987997 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":69,"spinlock_wait_cycles":10240,"thread_start_us":105,"threads_started":1}
I20260808 08:20:16.989120 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.979484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47632)
0808 08:20:16.979670 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.979684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.979778 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.980098 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.980102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.980315 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:16.980410 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.980415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.981131 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.981136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.981689 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.981698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.982732 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.982735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.982859 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.983323 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:16.983342 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:16.985465 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:16.987479 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.987482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.987484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.987741 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.988044 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:16.988047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:16.988049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:16.988137 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:16.988570 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:16.988573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:16.988682 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 08:20:16.988810 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 08:20:16.988974 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 08:20:16.989809 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.979529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.12.13.60:36387)
0808 08:20:16.979643 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.979647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.979727 (+ 80us) server_negotiation.cc:407] Connection header received
0808 08:20:16.979939 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.979941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.979989 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.980079 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:16.980523 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.981022 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.981814 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.982492 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.982602 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.985610 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:16.985614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:16.985619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:16.985646 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:16.987358 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.987852 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.987855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.987857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.987937 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:16.988236 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:16.988242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:16.988246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:16.988461 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:16.988576 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:16.989625 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0808 08:20:16.989712 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 08:20:16.990895 21569 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.13.60:36387: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
member_type: NON_VOTER
last_known_addr {
host: "127.12.13.60"
port: 36387
}
attrs {
promote: true
}
}
dest_uuid: "a78fd6a525bd4b3e942061d1d81d610e"
cas_config_opid_index: 5
I20260808 08:20:16.991586 21569 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } }
I20260808 08:20:16.993016 22449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:16.994560 22449 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.13.60:36387
I20260808 08:20:16.994764 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:16.994887 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:16.994879 22659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:16.995369 22471 master.cc:537] Master@127.12.13.60:36387 shutting down...
I20260808 08:20:16.997104 22659 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Refusing update from remote peer a78fd6a525bd4b3e942061d1d81d610e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 08:20:16.997992 22450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 08:20:17.000028 21550 consensus_peers.cc:487] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e -> Peer d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60:36387): Couldn't send request to peer d6e9d8a05b014a7691b9c07e4fd32053. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 08:20:17.000741 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.992873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56218)
0808 08:20:16.994537 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:16.994558 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:16.995037 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:16.995640 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:16.995645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:16.995669 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:16.995798 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.995808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.997059 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.997065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:16.998785 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:16.998801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.000087 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.000169 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.000302 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.000361 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.000450 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.000556 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511}
I20260808 08:20:17.001231 22471 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:17.001366 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:16.992954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.12.13.61:46093)
0808 08:20:16.993103 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:16.993108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:16.994679 (+ 1571us) server_negotiation.cc:407] Connection header received
0808 08:20:16.995193 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:16.995226 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:16.995313 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:16.995422 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:16.996009 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.996905 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.998966 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:16.999691 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:16.999772 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:16.999984 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.001032 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.001136 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:20:17.001547 22471 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:17.001668 22471 tablet_replica.cc:330] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: stopping tablet replica
I20260808 08:20:17.002528 22527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 LEARNER]: Refusing update from remote peer a78fd6a525bd4b3e942061d1d81d610e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 08:20:17.003032 22450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 08:20:17.004966 22527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.006908 22570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78fd6a525bd4b3e942061d1d81d610e. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.007025 22570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: LEARNER
I20260808 08:20:17.007102 22570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: NON_VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: true } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.007176 22570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: LEARNER
I20260808 08:20:17.007758 22471 master.cc:552] Master@127.12.13.60:36387 shutdown complete.
I20260808 08:20:17.007849 22471 master_runner.cc:305] Clearing existing system tablet
I20260808 08:20:17.008147 22471 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:17.008239 22471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:17.008710 22450 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: attempting to promote NON_VOTER 5ff3ab74c44b4b3398655f227749a56e to VOTER
I20260808 08:20:17.009195 22450 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } }
I20260808 08:20:17.010444 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.010126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47646)
0808 08:20:17.010276 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.010362 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 08:20:17.010648 22527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 LEARNER]: Refusing update from remote peer a78fd6a525bd4b3e942061d1d81d610e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 08:20:17.011047 22449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 08:20:17.011500 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.011048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47658)
0808 08:20:17.011374 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.011423 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
W20260808 08:20:17.011660 21550 consensus_peers.cc:487] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e -> Peer d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60:36387): Couldn't send request to peer d6e9d8a05b014a7691b9c07e4fd32053. Status: Network error: Client connection negotiation failed: client connection to 127.12.13.60:36387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 08:20:17.012466 22471 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260808 08:20:17.012704 22449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5ff3ab74c44b4b3398655f227749a56e (127.12.13.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.013703 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.013967 22471 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:17.014079 22471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:17.014114 22471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "11d535b959ffd58f1f4491d85b560755"
server_key_iv: "6d03ae6d9e196c9cda225618534a2342"
server_key_version: "encryptionkey@0"
I20260808 08:20:17.014329 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.014438 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:17.014557 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.014637 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:17.015343 22527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5ff3ab74c44b4b3398655f227749a56e (127.12.13.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.016413 22471 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:17.017047 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.017151 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:17.018851 22471 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 08:20:17.018932 22471 log.cc:1196] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 08:20:17.019181 22471 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Deleting consensus metadata
I20260808 08:20:17.019346 22471 master_runner.cc:315] Copying system tablet from 127.12.13.62:33243
I20260808 08:20:17.020574 22471 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: tablet copy: Beginning tablet copy session from remote peer at address 127.12.13.62:33243
I20260808 08:20:17.030656 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.021423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.12.13.62:33243)
0808 08:20:17.021568 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.021573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.021906 (+ 333us) server_negotiation.cc:407] Connection header received
0808 08:20:17.022073 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.022076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.022131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.022218 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.022776 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.023574 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.024761 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.025094 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.025200 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.026749 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.026754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.026757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.026785 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.028566 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.029086 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.029089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.029091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.029188 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.029596 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.029600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.029602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.029861 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.029966 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.030349 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.030482 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:20:17.030668 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.021496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41262)
0808 08:20:17.021780 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.021795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.021949 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.022256 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.022260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.022510 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.022646 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.022654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.023678 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.023681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.024623 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.024634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.025223 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.025226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.025323 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.026124 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.026139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.026591 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.028695 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.028698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.028701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.028977 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.029293 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.029296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.029298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.029406 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.030050 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.030053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.030157 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.030264 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.030453 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
I20260808 08:20:17.031834 21604 tablet_copy_service.cc:140] P a78fd6a525bd4b3e942061d1d81d610e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d6e9d8a05b014a7691b9c07e4fd32053 ({username='oryx', principal='oryx/127.12.13.60@KRBTEST.COM'} at 127.0.0.1:41262)
I20260808 08:20:17.031960 21604 tablet_copy_service.cc:161] P a78fd6a525bd4b3e942061d1d81d610e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d6e9d8a05b014a7691b9c07e4fd32053 at {username='oryx', principal='oryx/127.12.13.60@KRBTEST.COM'} at 127.0.0.1:41262: session id = d6e9d8a05b014a7691b9c07e4fd32053-00000000000000000000000000000000
I20260808 08:20:17.033123 21604 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 08:20:17.034647 22471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 08:20:17.037252 22471 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: tablet copy: Starting download of 0 data blocks...
I20260808 08:20:17.037487 22471 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: tablet copy: Starting download of 1 WAL segments...
I20260808 08:20:17.039299 22471 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 08:20:17.041468 21604 tablet_copy_service.cc:342] P a78fd6a525bd4b3e942061d1d81d610e: Request end of tablet copy session d6e9d8a05b014a7691b9c07e4fd32053-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.13.60@KRBTEST.COM'} at 127.0.0.1:41262
I20260808 08:20:17.041577 21604 tablet_copy_service.cc:433] P a78fd6a525bd4b3e942061d1d81d610e: ending tablet copy session d6e9d8a05b014a7691b9c07e4fd32053-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d6e9d8a05b014a7691b9c07e4fd32053
I20260808 08:20:17.050012 22471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:17.051923 22763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:17.052006 22760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:17.052886 22471 server_base.cc:1034] running on GCE node
W20260808 08:20:17.054453 22761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:17.058215 22471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:17.058625 22471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:17.059782 22471 hybrid_clock.cc:648] HybridClock initialized: now 1786177217059752 us; error 59 us; skew 500 ppm
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:17.061553 22471 init.cc:377] Logged in from keytab as oryx/127.12.13.60@KRBTEST.COM (short username oryx)
I20260808 08:20:17.062129 22471 webserver.cc:460] Webserver started at http://127.12.13.60:45109/ using document root <none> and password file <none>
I20260808 08:20:17.062333 22471 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:17.062381 22471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:17.063814 22471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:17.064968 22773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.065274 22471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:17.065353 22471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "11d535b959ffd58f1f4491d85b560755"
server_key_iv: "6d03ae6d9e196c9cda225618534a2342"
server_key_version: "encryptionkey@0"
I20260808 08:20:17.065438 22471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:17.079473 22088 heartbeater.cc:503] Master 127.12.13.62:33243 was elected leader, sending a full tablet report...
I20260808 08:20:17.095809 22471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:17.098760 22471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:17.104789 22471 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.60:36387
I20260808 08:20:17.104821 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.60:36387 every 8 connection(s)
I20260808 08:20:17.105461 22471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
I20260808 08:20:17.107535 22862 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:20:17.108763 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Bootstrap starting.
I20260808 08:20:17.121416 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:20:17.121798 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: Bootstrap complete.
I20260808 08:20:17.122532 22862 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } }
I20260808 08:20:17.122684 22862 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6e9d8a05b014a7691b9c07e4fd32053, State: Initialized, Role: LEARNER
I20260808 08:20:17.122798 22862 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } }
I20260808 08:20:17.123270 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.123355 22862 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:17.123503 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: This master's current role is: LEARNER
I20260808 08:20:17.128769 22883 catalog_manager.cc:1259] Loaded cluster ID: dd7d30563e8340cbb42cf8185b0a59cb
I20260808 08:20:17.128854 22883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: loading cluster ID for follower catalog manager: success
I20260808 08:20:17.131237 22883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: acquiring CA information for follower catalog manager: success
I20260808 08:20:17.132858 22883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 08:20:17.143556 22740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:17.143853 22740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:17.143900 22740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:17.148334 22740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:17.148414 22740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:17.148478 22740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:17.154100 22740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:38235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=44709
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:17.155351 22740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:17.156513 22740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:17.167330 22893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:17.167353 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:17.167419 22897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:17.167843 22740 server_base.cc:1034] running on GCE node
I20260808 08:20:17.168286 22740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:17.168946 22740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:17.170113 22740 hybrid_clock.cc:648] HybridClock initialized: now 1786177217170077 us; error 47 us; skew 500 ppm
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:17.173163 22740 init.cc:377] Logged in from keytab as oryx/127.12.13.1@KRBTEST.COM (short username oryx)
I20260808 08:20:17.174129 22740 webserver.cc:460] Webserver started at http://127.12.13.1:44709/ using document root <none> and password file <none>
I20260808 08:20:17.174705 22740 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:17.174782 22740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:17.178891 22740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:20:17.181178 22906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.182657 22740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:17.182801 22740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "307ad2ac0a53478a8deda7ad2f09a713"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "2e79ecfd245ceddad60c482e88c814a9"
server_key_iv: "203264c837edc4fb9bbe9437b7920add"
server_key_version: "encryptionkey@0"
I20260808 08:20:17.183410 22740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:17.194659 22740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:17.198412 22740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:17.198630 22740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:17.199291 22740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:17.200250 22740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:17.200300 22740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.200449 22740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:17.200505 22740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.210469 22740 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:38235
I20260808 08:20:17.210494 23066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:38235 every 8 connection(s)
I20260808 08:20:17.211598 22740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:17.221055 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 22740
I20260808 08:20:17.221303 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 21896
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.60@KRBTEST.COM
I20260808 08:20:17.232283 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.213592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40905 (local address 127.12.13.60:36387)
0808 08:20:17.213858 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.213894 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.215707 (+ 1813us) server_negotiation.cc:407] Connection header received
0808 08:20:17.216552 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.216555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.216613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.216714 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.217802 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.218562 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220253 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.220591 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220717 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.227649 (+ 6932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.227656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.227662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.227698 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.230095 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.230608 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.230612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.230614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.230709 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.231043 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.231046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.231048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.231259 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.231374 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.231952 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.232095 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260808 08:20:17.232386 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.215730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40621 (local address 127.12.13.62:33243)
0808 08:20:17.215905 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.215910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.215926 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:20:17.216283 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.216286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.216339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.216419 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.217807 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.218639 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220286 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.220612 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.227649 (+ 6928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.227655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.227662 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.227700 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.229546 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.230384 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.230391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.230395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.230522 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.231060 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.231066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.231071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.231394 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.231482 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.231973 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.232127 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 08:20:17.233407 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.215329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.1:40621)
0808 08:20:17.215734 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.215748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.216004 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.216497 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.216505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.217078 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.217638 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.217645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.218751 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.218753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.220125 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.220137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.220887 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.220986 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.224017 (+ 3031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.224052 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.227422 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.229698 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.229704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.229723 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.230153 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.230669 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.230675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.230680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.230928 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.231564 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.231570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.231696 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.232629 (+ 933us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.232807 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":309,"spinlock_wait_cycles":45824,"thread_start_us":93,"threads_started":1}
I20260808 08:20:17.233469 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.214169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.12.13.1:40905)
0808 08:20:17.215581 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.215594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.215931 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.216746 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.216749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.217074 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.217636 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.217645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.218747 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.218751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.220125 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.220137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.220730 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.220854 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.224015 (+ 3161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.224040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.227407 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.230237 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.230243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.230247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.230500 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.230806 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.230810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.230812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.230938 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.231393 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.231401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.231693 (+ 292us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.231906 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.232265 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"mutex_wait_us":314,"spinlock_wait_cycles":92416,"thread_start_us":160,"threads_started":1}
I20260808 08:20:17.235036 23068 heartbeater.cc:347] Connected to a master server at 127.12.13.60:36387
I20260808 08:20:17.235064 23069 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33243
I20260808 08:20:17.235314 23069 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.235314 23068 heartbeater.cc:464] Registering TS with master...
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
I20260808 08:20:17.235984 23069 heartbeater.cc:511] Master 127.12.13.62:33243 requested a full tablet report, sending...
I20260808 08:20:17.235984 23068 heartbeater.cc:511] Master 127.12.13.60:36387 requested a full tablet report, sending...
I20260808 08:20:17.237785 21569 ts_manager.cc:194] Re-registered known tserver with Master: 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235)
I20260808 08:20:17.237793 22789 ts_manager.cc:194] Registered new tserver with Master: 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235)
I20260808 08:20:17.238656 21569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:40621
I20260808 08:20:17.240833 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:41789
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:33243
--webserver_port=43747
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:20:17.241294 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.213687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:47899 (local address 127.12.13.61:46093)
0808 08:20:17.213895 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.213900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.214901 (+ 1001us) server_negotiation.cc:407] Connection header received
0808 08:20:17.216153 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.216156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.216220 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.216308 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.217772 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.218639 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220052 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.220341 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220451 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.236310 (+ 15859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.236317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.236324 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.236378 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.238156 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.238663 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.238670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.238675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.238768 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.240373 (+ 1605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.240379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.240384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.240635 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.240756 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.241025 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.241143 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:20:17.241317 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.214169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.12.13.1:47899)
0808 08:20:17.214711 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.214766 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.215914 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.216499 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.216505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.217042 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.217615 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.217629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.218796 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.218799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.219916 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.219926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.220512 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.220515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.233976 (+ 13461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.234457 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.234476 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.236093 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.238281 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.238284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.238286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.238529 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.239017 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.239023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.239028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.239120 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.240744 (+ 1624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.240747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.240854 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.240970 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.241135 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":166,"thread_start_us":183,"threads_started":1}
I20260808 08:20:17.242277 23071 heartbeater.cc:347] Connected to a master server at 127.12.13.61:46093
I20260808 08:20:17.242393 23071 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.242636 23071 heartbeater.cc:511] Master 127.12.13.61:46093 requested a full tablet report, sending...
I20260808 08:20:17.243877 22512 ts_manager.cc:194] Registered new tserver with Master: 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235)
I20260808 08:20:17.314280 22294 heartbeater.cc:503] Master 127.12.13.62:33243 was elected leader, sending a full tablet report...
W20260808 08:20:17.364238 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:17.364643 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:17.364735 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:17.369165 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:17.369258 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:17.369345 23084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:17.374917 23084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:41789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=43747
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:17.376073 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:17.377172 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:20:17.388484 23084 server_base.cc:1034] running on GCE node
W20260808 08:20:17.388434 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:17.388424 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:17.388427 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:17.389218 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:17.389953 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:17.391158 23084 hybrid_clock.cc:648] HybridClock initialized: now 1786177217391142 us; error 35 us; skew 500 ppm
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:17.394091 23084 init.cc:377] Logged in from keytab as oryx/127.12.13.2@KRBTEST.COM (short username oryx)
I20260808 08:20:17.395180 23084 webserver.cc:460] Webserver started at http://127.12.13.2:43747/ using document root <none> and password file <none>
I20260808 08:20:17.395746 23084 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:17.395826 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:17.399905 23084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:20:17.402171 23103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.403496 23084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:17.403601 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "948827c379784a05be72ff83b08d296d"
format_stamp: "Formatted at 2026-08-08 08:20:15 on dist-test-slave-6ntk"
server_key: "87639643bc2a4bb5f3a786c86c81a50d"
server_key_iv: "ae43e7895886a70a34b869dfaf7ddde6"
server_key_version: "encryptionkey@0"
I20260808 08:20:17.404126 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:17.428088 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:17.431680 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:17.431849 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:17.432456 23084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:17.433320 23084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:17.433378 23084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.433420 23084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:17.433436 23084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.443028 23084 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:41789
I20260808 08:20:17.443049 23275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:41789 every 8 connection(s)
I20260808 08:20:17.444207 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:17.449705 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 23084
I20260808 08:20:17.449954 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 22096
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.60@KRBTEST.COM
I20260808 08:20:17.462057 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.446127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:60717 (local address 127.12.13.62:33243)
0808 08:20:17.446269 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.446274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.447271 (+ 997us) server_negotiation.cc:407] Connection header received
0808 08:20:17.448534 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.448538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.448594 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.448650 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.450331 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.451069 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.452668 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.453034 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.453115 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.456810 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.456817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.456823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.456879 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.459083 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.459824 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.459841 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.459846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.459941 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.460390 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.460394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.460395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.460663 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.460759 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.461678 (+ 919us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.461846 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 08:20:17.462057 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.447783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:50105 (local address 127.12.13.60:36387)
0808 08:20:17.447927 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.447930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.447948 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:20:17.448534 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.448537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.448594 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.448650 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.452519 (+ 3869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.453288 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.454535 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.454907 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.454996 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.457667 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.457673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.457678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.457719 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.459564 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.460063 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.460067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.460069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.460156 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.460491 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.460494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.460495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.460697 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.460763 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.461684 (+ 921us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.461839 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 08:20:17.464124 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.447632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.12.13.2:50105)
0808 08:20:17.447884 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.447898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.448296 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.448874 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.448882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.449397 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.452345 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.452357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.453441 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.453444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.454375 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.454386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.455197 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.455203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.455300 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.455922 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.455943 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.457453 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.459697 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.459700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.459702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.459942 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.460272 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.460274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.460276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.460382 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.460843 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.460851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.461344 (+ 493us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.462340 (+ 996us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.462889 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2463,"spinlock_wait_cycles":27520,"thread_start_us":90,"threads_started":1}
I20260808 08:20:17.464124 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.446610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:60717)
0808 08:20:17.447107 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.447161 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.448296 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.448874 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.448882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.449397 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.450157 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.450178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.451237 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.451243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.452535 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.452543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.453342 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.453347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.453593 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.454381 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.454404 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.456562 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.459257 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.459266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.459289 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.459697 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.460057 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.460060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.460062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.460276 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.460844 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.460851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.461344 (+ 493us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.462330 (+ 986us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.462938 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":377,"spinlock_wait_cycles":40320,"thread_start_us":149,"threads_started":1}
I20260808 08:20:17.465658 23278 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33243
I20260808 08:20:17.465979 23278 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.466533 23276 heartbeater.cc:347] Connected to a master server at 127.12.13.60:36387
I20260808 08:20:17.466627 23276 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.466759 23278 heartbeater.cc:511] Master 127.12.13.62:33243 requested a full tablet report, sending...
I20260808 08:20:17.466811 23276 heartbeater.cc:511] Master 127.12.13.60:36387 requested a full tablet report, sending...
I20260808 08:20:17.468233 21569 ts_manager.cc:194] Re-registered known tserver with Master: 948827c379784a05be72ff83b08d296d (127.12.13.2:41789)
I20260808 08:20:17.468997 21569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:60717
I20260808 08:20:17.473613 22789 ts_manager.cc:194] Registered new tserver with Master: 948827c379784a05be72ff83b08d296d (127.12.13.2:41789)
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
I20260808 08:20:17.481256 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:44389
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:33243
--webserver_port=34869
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:20:17.482375 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.446148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:59287 (local address 127.12.13.61:46093)
0808 08:20:17.446307 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.446313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.447268 (+ 955us) server_negotiation.cc:407] Connection header received
0808 08:20:17.448518 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.448522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.448584 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.448695 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.474658 (+ 25963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.475445 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.476289 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.476528 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.476652 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.478955 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.478959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.478962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.478993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.480456 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.480992 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.480995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.480997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.481082 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.481435 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.481440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.481443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.481660 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.481743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.482124 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.482228 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:20:17.482913 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.446610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.12.13.2:59287)
0808 08:20:17.447106 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.447162 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.448294 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.448873 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.448880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.449287 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.474461 (+ 25174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.474478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.475583 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.475586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.476160 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.476167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.476685 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.476689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.476806 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.477377 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.477389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.478807 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.480584 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.480587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.480590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.480865 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.481179 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.481185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.481190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.481294 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.481769 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.481775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.481939 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.482056 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.482254 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":192,"spinlock_wait_cycles":6784,"thread_start_us":112,"threads_started":1}
I20260808 08:20:17.483743 23279 heartbeater.cc:347] Connected to a master server at 127.12.13.61:46093
I20260808 08:20:17.483865 23279 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.484151 23279 heartbeater.cc:511] Master 127.12.13.61:46093 requested a full tablet report, sending...
I20260808 08:20:17.484944 22512 ts_manager.cc:194] Registered new tserver with Master: 948827c379784a05be72ff83b08d296d (127.12.13.2:41789)
I20260808 08:20:17.606933 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.602809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.12.13.60:36387)
0808 08:20:17.602943 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.602950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.603151 (+ 201us) server_negotiation.cc:407] Connection header received
0808 08:20:17.603343 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.603346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.603403 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.603493 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:17.603815 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.604364 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.605977 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.606405 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.606494 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.606617 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.606764 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.606820 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 08:20:17.607019 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.602711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47662)
0808 08:20:17.603040 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.603057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.603202 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.603533 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.603538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.603558 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:17.603681 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.603686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.604490 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.604493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.605794 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.605806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.606533 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.606537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.606635 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.606665 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.606764 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.606841 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":86,"threads_started":1}
I20260808 08:20:17.607908 22803 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 08:20:17.608362 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78fd6a525bd4b3e942061d1d81d610e. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.608546 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: This master's current role is: LEARNER
W20260808 08:20:17.608351 23286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:17.608721 23286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:17.608834 23286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:17.613021 23286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:17.613088 23286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:17.613188 23286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:17.619153 23286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:44389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=34869
--tserver_master_addrs=127.12.13.62:33243,127.12.13.61:46093,127.12.13.60:36387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:17.620452 23286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:17.621608 23286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:17.633189 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:17.633301 23294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:17.633431 23286 server_base.cc:1034] running on GCE node
W20260808 08:20:17.633224 23297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:17.634156 23286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:17.634889 23286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:17.636059 23286 hybrid_clock.cc:648] HybridClock initialized: now 1786177217636026 us; error 48 us; skew 500 ppm
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:17.639317 23286 init.cc:377] Logged in from keytab as oryx/127.12.13.3@KRBTEST.COM (short username oryx)
I20260808 08:20:17.640283 23286 webserver.cc:460] Webserver started at http://127.12.13.3:34869/ using document root <none> and password file <none>
I20260808 08:20:17.640978 23286 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:17.641064 23286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:17.645264 23286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:17.647571 23311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.648880 23286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:17.648996 23286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "930622da90e04994b43dc0f72fafdcba"
format_stamp: "Formatted at 2026-08-08 08:20:16 on dist-test-slave-6ntk"
server_key: "ebdd7ecb7907808a770432ffd72b277c"
server_key_iv: "a139876625c2b91c9de78d441848f19b"
server_key_version: "encryptionkey@0"
I20260808 08:20:17.649519 23286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:17.662568 23286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:17.666529 23286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:17.666771 23286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:17.667420 23286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:17.668354 23286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:17.668403 23286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.668447 23286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:17.668462 23286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:17.678390 23286 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:44389
I20260808 08:20:17.678478 23479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:44389 every 8 connection(s)
I20260808 08:20:17.679615 23286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:17.679869 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 23286
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.61@KRBTEST.COM
Aug 08 08:20:17 dist-test-slave-6ntk krb5kdc[21506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177217, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.60@KRBTEST.COM
I20260808 08:20:17.697887 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.681710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:42137 (local address 127.12.13.62:33243)
0808 08:20:17.682032 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.682038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.683228 (+ 1190us) server_negotiation.cc:407] Connection header received
0808 08:20:17.684558 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.684561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.684623 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.684681 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.686662 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.687372 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.689094 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.689380 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.689498 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.692711 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.692715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.692718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.692746 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.695150 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.695804 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.695810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.695815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.695927 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.696401 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.696407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.696411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.696654 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.696746 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.697517 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.697650 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":197}
I20260808 08:20:17.698946 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.682459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.12.13.62:33243)
0808 08:20:17.684716 (+ 2257us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.684730 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.684742 (+ 12us) server_negotiation.cc:407] Connection header received
0808 08:20:17.684778 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.684780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.684816 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.684895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.686512 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.687368 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.689093 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.689911 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.690018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.693550 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.693556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.693561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.693599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.696871 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.697357 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.697361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.697362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.697457 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.697755 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.697758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.697760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.697994 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.698120 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.698519 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.698699 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":2186,"thread_start_us":93,"threads_started":1}
I20260808 08:20:17.699703 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.682262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:42137)
0808 08:20:17.683074 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.683128 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.684299 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.685127 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.685138 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.685646 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.686505 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.686525 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.687552 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.687555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.688952 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.688962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.689554 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.689558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.689766 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.690427 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.690451 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.692525 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.695284 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.695291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.695309 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.695679 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.696059 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.696063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.696068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.696275 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.696820 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.696830 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.697213 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.698137 (+ 924us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.698522 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":4,"thread_start_us":153,"threads_started":1}
I20260808 08:20:17.700338 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.682748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.12.13.61:46093)
0808 08:20:17.682892 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.682899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.682961 (+ 62us) server_negotiation.cc:407] Connection header received
0808 08:20:17.683236 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.683240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.683314 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.683386 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.684749 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.685642 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.687102 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.687526 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.687629 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.694205 (+ 6576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.694213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.694216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.694246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.696291 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.696867 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.696871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.696873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.696970 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.699280 (+ 2310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.699283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.699285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.699512 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.699614 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.700057 (+ 443us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.700193 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:20:17.700992 23484 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33243
I20260808 08:20:17.701251 23484 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.701844 23484 heartbeater.cc:511] Master 127.12.13.62:33243 requested a full tablet report, sending...
I20260808 08:20:17.703408 21567 ts_manager.cc:194] Re-registered known tserver with Master: 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:17.704236 21567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:42137
I20260808 08:20:17.704272 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.683406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.12.13.3:58829)
0808 08:20:17.684823 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.684839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.684937 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.686208 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.686210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.686418 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.687623 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.687629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.691654 (+ 4025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.691657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.692742 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.692757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.693538 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.693542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.693634 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.695003 (+ 1369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.695021 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.696888 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.702295 (+ 5407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.702302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.702306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.702625 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.703051 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.703058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.703062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.703163 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.703699 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.703703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.703815 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.703919 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.704061 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":1044,"thread_start_us":78,"threads_started":1}
I20260808 08:20:17.705098 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.683537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.12.13.60:36387)
0808 08:20:17.683856 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.683862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.686786 (+ 2924us) server_negotiation.cc:407] Connection header received
0808 08:20:17.686984 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.686991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.687058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.687117 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.688345 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.689254 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.690678 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.691028 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.691111 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.698919 (+ 7808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.698923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.698925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.698953 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.703136 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.703704 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.703707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.703709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.703797 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.704111 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.704114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.704116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.704320 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.704436 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.704805 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.704952 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260808 08:20:17.704952 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.683610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:58829 (local address 127.12.13.60:36387)
0808 08:20:17.684237 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.684242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.685836 (+ 1594us) server_negotiation.cc:407] Connection header received
0808 08:20:17.685949 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.685952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.686005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.686094 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.690669 (+ 4575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.691499 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.692935 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.693383 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.693849 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.697376 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.697383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.697388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.697421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.702122 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.702794 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.702798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.702800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.702903 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.703292 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.703296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.703297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.703543 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.704534 (+ 991us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.704621 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.704771 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":102,"threads_started":1}
I20260808 08:20:17.705842 23482 heartbeater.cc:347] Connected to a master server at 127.12.13.60:36387
I20260808 08:20:17.705978 23482 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.706115 23482 heartbeater.cc:511] Master 127.12.13.60:36387 requested a full tablet report, sending...
I20260808 08:20:17.707007 22789 ts_manager.cc:194] Registered new tserver with Master: 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:17.703727 21569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:17.708900 21569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:17.712059 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: NON_VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: true } } }
I20260808 08:20:17.712208 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:17.714118 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.682262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.12.13.3:54017)
0808 08:20:17.685290 (+ 3028us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.685304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.685400 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.685801 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.685805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.686113 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:17.688320 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.688333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.689498 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.689501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.690517 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.690528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.693379 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.693383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.693516 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.694104 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:17.694122 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:17.697306 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:17.711594 (+ 14288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.711598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.711600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.711924 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.712354 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:17.712362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:17.712367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:17.712481 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:17.713020 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:17.713023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:17.713174 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.713292 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.713481 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2929,"mutex_wait_us":3378,"thread_start_us":128,"threads_started":1}
I20260808 08:20:17.714876 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.683697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:54017 (local address 127.12.13.61:46093)
0808 08:20:17.685142 (+ 1445us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.685146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.685478 (+ 332us) server_negotiation.cc:407] Connection header received
0808 08:20:17.685519 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.685522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.685566 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.685644 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.688468 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.689344 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.692407 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.692758 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.692835 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.697458 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.697465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.697472 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.697498 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.711419 (+ 13921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.712086 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.712091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.712093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.712215 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.712618 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.712621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.712623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.712892 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.714459 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.714547 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.714707 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":86,"threads_started":1}
I20260808 08:20:17.715545 23486 heartbeater.cc:347] Connected to a master server at 127.12.13.61:46093
I20260808 08:20:17.715669 23486 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:17.715906 23486 heartbeater.cc:511] Master 127.12.13.61:46093 requested a full tablet report, sending...
I20260808 08:20:17.716701 22512 ts_manager.cc:194] Registered new tserver with Master: 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:17.717530 22748 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: attempting to promote NON_VOTER d6e9d8a05b014a7691b9c07e4fd32053 to VOTER
I20260808 08:20:17.717839 22748 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:17.718963 22803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 LEARNER]: Refusing update from remote peer a78fd6a525bd4b3e942061d1d81d610e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 08:20:17.719352 22527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Refusing update from remote peer a78fd6a525bd4b3e942061d1d81d610e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 08:20:17.719436 23505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 08:20:17.719745 23505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 08:20:17.721964 23505 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.723660 22527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.724196 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.724298 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:17.724382 23505 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.724509 23505 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:17.724401 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.724665 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:17.725641 22803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.726063 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.726188 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:17.727768 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a78fd6a525bd4b3e942061d1d81d610e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:17.727912 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:17.737443 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.730884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38235 (local address 127.0.0.1:47538)
0808 08:20:17.731220 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.731237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.731318 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.733246 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.733249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.733265 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:17.733341 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.733347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.734845 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.734852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.735992 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.736003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.736911 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.736919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.737126 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.737155 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.737217 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.737294 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":74}
I20260808 08:20:17.737514 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.731210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.12.13.1:38235)
0808 08:20:17.732844 (+ 1634us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.732859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.732880 (+ 21us) server_negotiation.cc:407] Connection header received
0808 08:20:17.732959 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.732964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.733123 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.733251 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:17.733478 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.734727 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.736131 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.736762 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.737010 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.737194 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.737257 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.737372 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1542,"thread_start_us":109,"threads_started":1}
I20260808 08:20:17.737818 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.732070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.12.13.2:41789)
0808 08:20:17.732464 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.732478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.732504 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:20:17.732591 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.732597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.732802 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.732989 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:17.733189 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.734323 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.735829 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.736663 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.736780 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.736936 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.737523 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.737641 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":144,"threads_started":1}
I20260808 08:20:17.737452 23526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.731249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:41789 (local address 127.0.0.1:47394)
0808 08:20:17.731508 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.731518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.731619 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.732949 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.732952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.732970 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:17.733093 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.733101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.734463 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.734466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.735686 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.735695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.736820 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.736828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.737077 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.737104 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.737188 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.737275 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":59,"thread_start_us":127,"threads_started":1}
I20260808 08:20:17.738813 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.731638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44389 (local address 127.0.0.1:56966)
0808 08:20:17.733504 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.733523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.733638 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.734326 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.734329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.734351 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:17.734446 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.734452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.735828 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.735832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.737113 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.737124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.738407 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.738412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.738511 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.738537 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.738616 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.738689 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"mutex_wait_us":62,"thread_start_us":108,"threads_started":1}
I20260808 08:20:17.741268 22978 tablet_service.cc:1467] Processing CreateTablet for tablet 29263f9206a74bacbdb6f8d9ba61d843 (DEFAULT_TABLE table=test-table [id=54e9fd36c0544e4a92b75d228800ee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:17.741281 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 29263f9206a74bacbdb6f8d9ba61d843 (DEFAULT_TABLE table=test-table [id=54e9fd36c0544e4a92b75d228800ee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:17.742249 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.731809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.12.13.3:44389)
0808 08:20:17.732599 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.732614 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.733641 (+ 1027us) server_negotiation.cc:407] Connection header received
0808 08:20:17.733759 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.733764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.734157 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.734313 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:17.734565 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.735698 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.737253 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.738100 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.738191 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.738336 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.738665 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.738767 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":139,"threads_started":1}
I20260808 08:20:17.742552 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29263f9206a74bacbdb6f8d9ba61d843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:17.743235 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29263f9206a74bacbdb6f8d9ba61d843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:17.745431 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 29263f9206a74bacbdb6f8d9ba61d843 (DEFAULT_TABLE table=test-table [id=54e9fd36c0544e4a92b75d228800ee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:17.746500 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29263f9206a74bacbdb6f8d9ba61d843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:17.751362 23533 tablet_bootstrap.cc:492] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Bootstrap starting.
I20260808 08:20:17.752192 23534 tablet_bootstrap.cc:492] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Bootstrap starting.
I20260808 08:20:17.754438 23534 tablet_bootstrap.cc:654] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:17.754506 23533 tablet_bootstrap.cc:654] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:17.754716 23536 tablet_bootstrap.cc:492] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Bootstrap starting.
I20260808 08:20:17.755367 23534 log.cc:824] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:17.755419 23533 log.cc:824] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:17.756879 23536 tablet_bootstrap.cc:654] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:17.757712 23536 log.cc:824] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:17.757783 23533 tablet_bootstrap.cc:492] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: No bootstrap required, opened a new log
I20260808 08:20:17.758033 23533 ts_tablet_manager.cc:1397] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 08:20:17.758991 23534 tablet_bootstrap.cc:492] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: No bootstrap required, opened a new log
I20260808 08:20:17.759212 23534 ts_tablet_manager.cc:1397] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 08:20:17.759832 23536 tablet_bootstrap.cc:492] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: No bootstrap required, opened a new log
I20260808 08:20:17.760022 23536 ts_tablet_manager.cc:1397] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:20:17.762230 23533 raft_consensus.cc:348] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.762530 23533 raft_consensus.cc:374] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:17.762593 23533 raft_consensus.cc:729] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307ad2ac0a53478a8deda7ad2f09a713, State: Initialized, Role: FOLLOWER
I20260808 08:20:17.762852 23534 raft_consensus.cc:348] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.763118 23534 raft_consensus.cc:374] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:17.763177 23534 raft_consensus.cc:729] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948827c379784a05be72ff83b08d296d, State: Initialized, Role: FOLLOWER
I20260808 08:20:17.763091 23533 consensus_queue.cc:260] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.763622 23534 consensus_queue.cc:260] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.764246 23533 ts_tablet_manager.cc:1428] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:20:17.764611 23069 heartbeater.cc:503] Master 127.12.13.62:33243 was elected leader, sending a full tablet report...
I20260808 08:20:17.764662 23534 ts_tablet_manager.cc:1428] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 08:20:17.764536 23536 raft_consensus.cc:348] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.764768 23536 raft_consensus.cc:374] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:17.764818 23536 raft_consensus.cc:729] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930622da90e04994b43dc0f72fafdcba, State: Initialized, Role: FOLLOWER
I20260808 08:20:17.765197 23536 consensus_queue.cc:260] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.766171 23536 ts_tablet_manager.cc:1428] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:20:17.767365 23484 heartbeater.cc:503] Master 127.12.13.62:33243 was elected leader, sending a full tablet report...
I20260808 08:20:17.768433 23278 heartbeater.cc:503] Master 127.12.13.62:33243 was elected leader, sending a full tablet report...
I20260808 08:20:17.799973 23542 raft_consensus.cc:482] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:17.800238 23542 raft_consensus.cc:504] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.801505 23542 leader_election.cc:288] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235), 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:17.805420 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.801952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:44309 (local address 127.12.13.1:38235)
0808 08:20:17.802080 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.802085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.802104 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:20:17.802253 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.802255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.802307 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.802528 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:17.802849 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.803342 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.804437 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.804822 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.804944 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.805013 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.805206 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.805274 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 08:20:17.805421 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.801798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38235 (local address 127.12.13.2:44309)
0808 08:20:17.801983 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.802000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.802114 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.802540 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.802544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.802562 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:17.802723 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.802729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.803466 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.803470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.804302 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.804311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.804950 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.804954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.805060 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.805087 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.805174 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.805252 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":82,"spinlock_wait_cycles":1792}
I20260808 08:20:17.805423 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.801969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:55085 (local address 127.12.13.3:44389)
0808 08:20:17.802092 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.802096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.802254 (+ 158us) server_negotiation.cc:407] Connection header received
0808 08:20:17.802402 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.802405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.802461 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.802547 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:17.802848 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.803343 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.804444 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.804847 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.804968 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.805031 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.805234 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.805298 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:20:17.805442 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.801798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44389 (local address 127.12.13.2:55085)
0808 08:20:17.802185 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:17.802199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:17.802281 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:17.802563 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:17.802566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:17.802585 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:17.802725 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.802731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.803463 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.803467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.804301 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:17.804310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.804976 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.804979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:17.805093 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.805116 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:20:17.805203 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 08:20:17.805280 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":243}
I20260808 08:20:17.806157 23005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29263f9206a74bacbdb6f8d9ba61d843" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307ad2ac0a53478a8deda7ad2f09a713" is_pre_election: true
I20260808 08:20:17.806205 23410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29263f9206a74bacbdb6f8d9ba61d843" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930622da90e04994b43dc0f72fafdcba" is_pre_election: true
I20260808 08:20:17.806524 23005 raft_consensus.cc:2454] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 0.
I20260808 08:20:17.806520 23410 raft_consensus.cc:2454] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 0.
I20260808 08:20:17.807024 23114 leader_election.cc:302] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 307ad2ac0a53478a8deda7ad2f09a713, 930622da90e04994b43dc0f72fafdcba, 948827c379784a05be72ff83b08d296d; no voters:
I20260808 08:20:17.807384 23542 raft_consensus.cc:2781] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:17.807487 23542 raft_consensus.cc:482] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:17.807559 23542 raft_consensus.cc:3037] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:17.808674 23542 raft_consensus.cc:504] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.809070 23542 leader_election.cc:288] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 election: Requested vote from peers 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235), 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:17.809520 23410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29263f9206a74bacbdb6f8d9ba61d843" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930622da90e04994b43dc0f72fafdcba"
I20260808 08:20:17.809669 23410 raft_consensus.cc:3037] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:17.809636 23005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29263f9206a74bacbdb6f8d9ba61d843" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307ad2ac0a53478a8deda7ad2f09a713"
I20260808 08:20:17.809813 23005 raft_consensus.cc:3037] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:17.810820 23410 raft_consensus.cc:2454] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 1.
I20260808 08:20:17.810864 23005 raft_consensus.cc:2454] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 1.
I20260808 08:20:17.811162 23111 leader_election.cc:302] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930622da90e04994b43dc0f72fafdcba, 948827c379784a05be72ff83b08d296d; no voters:
I20260808 08:20:17.811381 23542 raft_consensus.cc:2781] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:17.811645 23542 raft_consensus.cc:686] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 1 LEADER]: Becoming Leader. State: Replica: 948827c379784a05be72ff83b08d296d, State: Running, Role: LEADER
I20260808 08:20:17.812026 23542 consensus_queue.cc:237] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:17.815711 21567 catalog_manager.cc:5495] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948827c379784a05be72ff83b08d296d (127.12.13.2). New cstate: current_term: 1 leader_uuid: "948827c379784a05be72ff83b08d296d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:17.835376 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.831529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.12.13.2:41789)
0808 08:20:17.831665 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.831671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.831691 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:20:17.831899 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.831902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.831976 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.832094 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:17.832383 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.832886 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.833734 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.834058 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.834175 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.834511 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:17.834940 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:17.835082 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.835148 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.835208 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:20:17.841640 23410 raft_consensus.cc:1261] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Refusing update from remote peer 948827c379784a05be72ff83b08d296d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:17.841820 23005 raft_consensus.cc:1261] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 1 FOLLOWER]: Refusing update from remote peer 948827c379784a05be72ff83b08d296d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:17.842561 23544 consensus_queue.cc:1035] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [LEADER]: Connected to new peer: Peer: permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:17.842775 23542 consensus_queue.cc:1035] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [LEADER]: Connected to new peer: Peer: permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:17.851148 23547 mvcc.cc:205] Tried to move back new op lower bound from 7316181884270714880 to 7316181884160892928. Current Snapshot: MvccSnapshot[applied={T|T < 7316181884270714880}]
I20260808 08:20:17.854425 23553 mvcc.cc:205] Tried to move back new op lower bound from 7316181884270714880 to 7316181884160892928. Current Snapshot: MvccSnapshot[applied={T|T < 7316181884270714880}]
I20260808 08:20:17.864301 21567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:17.864465 21567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:17.869133 21567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a78fd6a525bd4b3e942061d1d81d610e: Sending DeleteTablet for 3 replicas of tablet 29263f9206a74bacbdb6f8d9ba61d843
I20260808 08:20:17.870566 22978 tablet_service.cc:1514] Processing DeleteTablet for tablet 29263f9206a74bacbdb6f8d9ba61d843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:17 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:47538
I20260808 08:20:17.870837 23180 tablet_service.cc:1514] Processing DeleteTablet for tablet 29263f9206a74bacbdb6f8d9ba61d843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:17 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:47394
I20260808 08:20:17.871176 23381 tablet_service.cc:1514] Processing DeleteTablet for tablet 29263f9206a74bacbdb6f8d9ba61d843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:17 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:56966
I20260808 08:20:17.871337 23563 tablet_replica.cc:330] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: stopping tablet replica
I20260808 08:20:17.871376 23564 tablet_replica.cc:330] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: stopping tablet replica
I20260808 08:20:17.871685 23563 raft_consensus.cc:2229] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:17.871754 23564 raft_consensus.cc:2229] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:17.872046 23563 raft_consensus.cc:2258] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:17.872290 23564 raft_consensus.cc:2258] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:17.873898 23563 ts_tablet_manager.cc:1905] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:17.874578 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 21525
I20260808 08:20:17.884405 23563 ts_tablet_manager.cc:1918] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:17.884541 23563 log.cc:1196] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/29263f9206a74bacbdb6f8d9ba61d843
I20260808 08:20:17.884940 23563 ts_tablet_manager.cc:1939] T 29263f9206a74bacbdb6f8d9ba61d843 P 307ad2ac0a53478a8deda7ad2f09a713: Deleting consensus metadata
I20260808 08:20:17.890223 23566 tablet_replica.cc:330] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: stopping tablet replica
I20260808 08:20:17.890704 23566 raft_consensus.cc:2229] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:17.891139 23566 raft_consensus.cc:2258] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:17.891592 23564 ts_tablet_manager.cc:1905] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:17.891765 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.873417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.12.13.61:46093)
0808 08:20:17.873631 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.873636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.873650 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:17.873687 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.873690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.873736 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.873823 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.882272 (+ 8449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.883122 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.884396 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.884757 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.884840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.887233 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.887240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.887246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.887285 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.889350 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.889856 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.889862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.889889 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.889989 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.890325 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.890341 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.890343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.890595 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.890727 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.891453 (+ 726us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.891588 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260808 08:20:17.893003 23566 ts_tablet_manager.cc:1905] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:17.894763 23564 ts_tablet_manager.cc:1918] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:17.894881 23564 log.cc:1196] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/29263f9206a74bacbdb6f8d9ba61d843
I20260808 08:20:17.895215 23564 ts_tablet_manager.cc:1939] T 29263f9206a74bacbdb6f8d9ba61d843 P 948827c379784a05be72ff83b08d296d: Deleting consensus metadata
I20260808 08:20:17.896023 23566 ts_tablet_manager.cc:1918] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:17.896168 23566 log.cc:1196] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/29263f9206a74bacbdb6f8d9ba61d843
I20260808 08:20:17.896564 23566 ts_tablet_manager.cc:1939] T 29263f9206a74bacbdb6f8d9ba61d843 P 930622da90e04994b43dc0f72fafdcba: Deleting consensus metadata
W20260808 08:20:17.898160 23312 connection.cc:535] server connection from 127.0.0.1:56966 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 08:20:17.898173 23111 connection.cc:535] server connection from 127.0.0.1:47394 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 08:20:17.898612 22908 connection.cc:535] server connection from 127.0.0.1:47538 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 08:20:17.903789 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.873516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.12.13.60:36387)
0808 08:20:17.874037 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.874041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.892936 (+ 18895us) server_negotiation.cc:407] Connection header received
0808 08:20:17.893149 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.893153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.893216 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.893279 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.894351 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.895201 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.896396 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.896818 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.896918 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.899588 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.899592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.899595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.899626 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.901282 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.901960 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.901966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.901967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.902067 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.902404 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.902406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.902408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.902652 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.903419 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.903499 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.903623 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
I20260808 08:20:17.911270 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.901540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.12.13.61:46093)
0808 08:20:17.901670 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.901675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.902185 (+ 510us) server_negotiation.cc:407] Connection header received
0808 08:20:17.902392 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.902396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.902450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.902505 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.903404 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.904120 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.905259 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.905598 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.905683 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.907477 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.907480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.907482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.907508 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.909364 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.909912 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.909917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.909919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.910010 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.910310 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.910313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.910315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.910557 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.910703 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.910935 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.911068 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 08:20:17.912678 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:17.902719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.12.13.60:36387)
0808 08:20:17.902827 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:17.902831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:17.903525 (+ 694us) server_negotiation.cc:407] Connection header received
0808 08:20:17.903680 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:17.903684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:17.903726 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:17.903773 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:17.904893 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.905600 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.906929 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:17.907320 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:17.907440 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:17.908969 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:17.908973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:17.908975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:17.909002 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:17.910726 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.911243 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.911246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.911248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.911338 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:17.911673 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:17.911677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:17.911679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:17.911895 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:17.912045 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:17.912414 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 08:20:17.912528 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 08:20:18.858254 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:18.857659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:49193)
0808 08:20:18.858004 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:18.858126 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
W20260808 08:20:18.858690 23278 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:33243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:18.859565 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:18.859071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.1:41107)
0808 08:20:18.859386 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:18.859480 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
W20260808 08:20:18.859877 23069 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:33243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:18.863657 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:18.863222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:60773)
0808 08:20:18.863488 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:18.863568 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260808 08:20:18.864078 23484 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:33243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:19.370064 23583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a78fd6a525bd4b3e942061d1d81d610e)
I20260808 08:20:19.370343 23583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:19.370571 23584 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a78fd6a525bd4b3e942061d1d81d610e)
I20260808 08:20:19.370787 23584 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:19.371591 23583 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243), d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60:36387)
I20260808 08:20:19.371747 23584 leader_election.cc:288] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243), 5ff3ab74c44b4b3398655f227749a56e (127.12.13.61:46093)
I20260808 08:20:19.372233 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.371753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41696)
0808 08:20:19.372013 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.372109 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
W20260808 08:20:19.372428 22490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:19.373056 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.371853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41700)
0808 08:20:19.372872 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.372948 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":85,"threads_started":1}
W20260808 08:20:19.373258 22778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:19.373487 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.373080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41704)
0808 08:20:19.373363 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.373418 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260808 08:20:19.373628 22490 leader_election.cc:334] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:19.374372 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.374113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41716)
0808 08:20:19.374226 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.374287 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 08:20:19.374516 22778 leader_election.cc:334] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:19.376345 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.372015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:46093 (local address 127.0.0.1:56264)
0808 08:20:19.372512 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.372578 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:19.372690 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:19.373019 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:19.373025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:19.373048 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:19.373180 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:19.373187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.374381 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.374387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:19.375281 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:19.375292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.375953 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.375956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:19.376032 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:19.376060 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 08:20:19.376123 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 08:20:19.376211 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":57,"threads_started":1}
I20260808 08:20:19.376396 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.372100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:36387 (local address 127.0.0.1:47696)
0808 08:20:19.372390 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.372408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:19.372524 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:19.373107 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:19.373111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:19.373134 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:19.373251 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:19.373258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.374342 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.374355 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:19.375311 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:19.375320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.375989 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.375992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:19.376080 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:19.376101 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 08:20:19.376182 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 08:20:19.376250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":45,"threads_started":1}
I20260808 08:20:19.376366 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.372518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.12.13.60:36387)
0808 08:20:19.372869 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:19.372873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:19.372886 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:20:19.372929 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:19.372931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:19.372974 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:19.373375 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:19.373475 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.374226 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.375454 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.375842 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.375974 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:19.376034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:19.376214 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 08:20:19.376277 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":68,"threads_started":1}
I20260808 08:20:19.377103 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.372299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.12.13.61:46093)
0808 08:20:19.372754 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:19.372759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:19.372773 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:19.372818 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:19.372821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:19.372876 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:19.373291 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:19.373391 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.374228 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.375409 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:19.375815 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:19.375908 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:19.376893 (+ 985us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:19.376935 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 08:20:19.377015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":92,"threads_started":1}
I20260808 08:20:19.377326 22803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ff3ab74c44b4b3398655f227749a56e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" is_pre_election: true
I20260808 08:20:19.377545 22803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff3ab74c44b4b3398655f227749a56e in term 1.
I20260808 08:20:19.377522 22527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5ff3ab74c44b4b3398655f227749a56e" is_pre_election: true
I20260808 08:20:19.377729 22527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e9d8a05b014a7691b9c07e4fd32053 in term 1.
I20260808 08:20:19.377970 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff3ab74c44b4b3398655f227749a56e, d6e9d8a05b014a7691b9c07e4fd32053; no voters: a78fd6a525bd4b3e942061d1d81d610e
I20260808 08:20:19.378149 22780 leader_election.cc:302] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff3ab74c44b4b3398655f227749a56e, d6e9d8a05b014a7691b9c07e4fd32053; no voters: a78fd6a525bd4b3e942061d1d81d610e
I20260808 08:20:19.378228 23583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 08:20:19.378288 23583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Starting leader election (detected failure of leader a78fd6a525bd4b3e942061d1d81d610e)
I20260808 08:20:19.378325 23583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 1 FOLLOWER]: Advancing to term 2
I20260808 08:20:19.378574 23584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 08:20:19.378651 23584 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a78fd6a525bd4b3e942061d1d81d610e)
I20260808 08:20:19.378707 23584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 1 FOLLOWER]: Advancing to term 2
I20260808 08:20:19.379464 23583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:19.379732 23584 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:19.379880 23583 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 2 election: Requested vote from peers a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243), d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60:36387)
I20260808 08:20:19.380177 23584 leader_election.cc:288] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [CANDIDATE]: Term 2 election: Requested vote from peers a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243), 5ff3ab74c44b4b3398655f227749a56e (127.12.13.61:46093)
I20260808 08:20:19.380492 22803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ff3ab74c44b4b3398655f227749a56e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
I20260808 08:20:19.380542 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.380100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41736)
0808 08:20:19.380408 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.380478 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260808 08:20:19.380509 22527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5ff3ab74c44b4b3398655f227749a56e"
I20260808 08:20:19.380615 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.380261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41742)
0808 08:20:19.380498 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.380554 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260808 08:20:19.380648 22803 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff3ab74c44b4b3398655f227749a56e in current term 2: Already voted for candidate d6e9d8a05b014a7691b9c07e4fd32053 in this term.
I20260808 08:20:19.380659 22527 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d6e9d8a05b014a7691b9c07e4fd32053 in current term 2: Already voted for candidate 5ff3ab74c44b4b3398655f227749a56e in this term.
I20260808 08:20:19.381357 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.381134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41744)
0808 08:20:19.381249 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.381302 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 08:20:19.381482 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.381276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41748)
0808 08:20:19.381352 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.381395 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 08:20:19.381598 22778 leader_election.cc:334] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
W20260808 08:20:19.381632 22490 leader_election.cc:334] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:19.381696 22778 leader_election.cc:302] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6e9d8a05b014a7691b9c07e4fd32053; no voters: 5ff3ab74c44b4b3398655f227749a56e, a78fd6a525bd4b3e942061d1d81d610e
I20260808 08:20:19.381755 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff3ab74c44b4b3398655f227749a56e; no voters: a78fd6a525bd4b3e942061d1d81d610e, d6e9d8a05b014a7691b9c07e4fd32053
I20260808 08:20:19.381847 23584 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 08:20:19.381940 23583 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 08:20:19.860085 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.859361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:58689)
0808 08:20:19.859745 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.859863 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260808 08:20:19.861061 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.860609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.1:55849)
0808 08:20:19.860927 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.860981 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260808 08:20:19.864987 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:19.864619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:60977)
0808 08:20:19.864849 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:19.864905 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260808 08:20:20.861780 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:20.861192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:48107)
0808 08:20:20.861548 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:20.861625 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":90,"threads_started":1}
I20260808 08:20:20.866304 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:20.865845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:58589)
0808 08:20:20.866156 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:20.866214 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":64,"threads_started":1}
I20260808 08:20:20.979069 23605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:20.979193 23605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:20.979630 23605 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243), d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60:36387)
I20260808 08:20:20.980284 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:20.979792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:41994)
0808 08:20:20.980052 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:20.980122 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260808 08:20:20.980384 22803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ff3ab74c44b4b3398655f227749a56e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" is_pre_election: true
I20260808 08:20:20.980562 22803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff3ab74c44b4b3398655f227749a56e in term 2.
I20260808 08:20:20.981087 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff3ab74c44b4b3398655f227749a56e, d6e9d8a05b014a7691b9c07e4fd32053; no voters:
I20260808 08:20:20.981171 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:20.980937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:42004)
0808 08:20:20.981050 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:20.981107 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 08:20:20.981248 22490 leader_election.cc:334] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:20.981273 23605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 08:20:20.981340 23605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:20.981407 23605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 2 FOLLOWER]: Advancing to term 3
I20260808 08:20:20.982569 23605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:20.982892 23605 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 3 election: Requested vote from peers a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243), d6e9d8a05b014a7691b9c07e4fd32053 (127.12.13.60:36387)
I20260808 08:20:20.983242 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:20.983026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:42018)
0808 08:20:20.983112 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:20.983167 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 08:20:20.983458 22803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ff3ab74c44b4b3398655f227749a56e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d6e9d8a05b014a7691b9c07e4fd32053"
I20260808 08:20:20.983611 22803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 2 FOLLOWER]: Advancing to term 3
I20260808 08:20:20.983883 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:20.983690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:42024)
0808 08:20:20.983785 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:20.983832 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 08:20:20.984055 22490 leader_election.cc:334] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
I20260808 08:20:20.984807 22803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff3ab74c44b4b3398655f227749a56e in term 3.
I20260808 08:20:20.985112 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff3ab74c44b4b3398655f227749a56e, d6e9d8a05b014a7691b9c07e4fd32053; no voters: a78fd6a525bd4b3e942061d1d81d610e
I20260808 08:20:20.985271 23605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 08:20:20.985430 23605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [term 3 LEADER]: Becoming Leader. State: Replica: 5ff3ab74c44b4b3398655f227749a56e, State: Running, Role: LEADER
I20260808 08:20:20.985711 23605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } }
I20260808 08:20:20.987288 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff3ab74c44b4b3398655f227749a56e. Latest consensus state: current_term: 3 leader_uuid: "5ff3ab74c44b4b3398655f227749a56e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:20.987397 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:20.987708 23615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:20.990216 23615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:20.990597 23615 catalog_manager.cc:1259] Loaded cluster ID: dd7d30563e8340cbb42cf8185b0a59cb
I20260808 08:20:20.990640 23615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:20.991156 23615 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:20.991528 23615 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Loaded TSK: 0
I20260808 08:20:20.992173 23615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:21.068354 22509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:21.071760 22509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:21.075197 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.074884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:42030)
0808 08:20:21.075037 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.075109 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 08:20:21.075702 22803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [term 3 FOLLOWER]: Refusing update from remote peer 5ff3ab74c44b4b3398655f227749a56e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 08:20:21.075966 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.075698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.0.0.1:42044)
0808 08:20:21.075850 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.075910 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 08:20:21.076124 22490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e -> Peer a78fd6a525bd4b3e942061d1d81d610e (127.12.13.62:33243): Couldn't send request to peer a78fd6a525bd4b3e942061d1d81d610e. Status: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 08:20:21.076447 23605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 08:20:21.078023 23617 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff3ab74c44b4b3398655f227749a56e. Latest consensus state: current_term: 3 leader_uuid: "5ff3ab74c44b4b3398655f227749a56e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:21.078171 23617 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:21.079391 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5ff3ab74c44b4b3398655f227749a56e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:21.079514 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:21.079870 23617 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5ff3ab74c44b4b3398655f227749a56e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a78fd6a525bd4b3e942061d1d81d610e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33243 } } peers { permanent_uuid: "5ff3ab74c44b4b3398655f227749a56e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "d6e9d8a05b014a7691b9c07e4fd32053" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 36387 } attrs { promote: false } } }
I20260808 08:20:21.080063 23617 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d8a05b014a7691b9c07e4fd32053 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:21.090137 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.085761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:41789 (local address 127.0.0.1:47410)
0808 08:20:21.085958 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.085973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:21.086055 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:21.086571 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:21.086575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:21.086593 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:21.086717 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:21.086723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.087871 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.087875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:21.088873 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:21.088882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.089603 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.089607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:21.089769 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.089811 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 08:20:21.089908 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 08:20:21.089984 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":40}
I20260808 08:20:21.090137 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.085973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.12.13.2:41789)
0808 08:20:21.086296 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:21.086303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:21.086320 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:20:21.086380 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:21.086383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:21.086434 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:21.086830 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:21.086962 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.087741 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.088997 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.089475 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.089582 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.089664 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:21.089927 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 08:20:21.090012 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":134,"threads_started":1}
I20260808 08:20:21.091053 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.086349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38235 (local address 127.0.0.1:47540)
0808 08:20:21.086624 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.086639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:21.086738 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:21.087307 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:21.087309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:21.087328 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:21.087437 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:21.087443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.088504 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.088507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:21.089721 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:21.089731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.090669 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.090675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:21.090766 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.090791 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 08:20:21.090839 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 08:20:21.090917 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":38,"thread_start_us":138,"threads_started":1}
I20260808 08:20:21.091449 23180 tablet_service.cc:1467] Processing CreateTablet for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be (DEFAULT_TABLE table=test-table [id=237065ac0eef469284d92fe2084ad110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:21.091768 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.086772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.12.13.1:38235)
0808 08:20:21.087065 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:21.087070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:21.087088 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:20:21.087145 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:21.087147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:21.087199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:21.087301 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:21.087574 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.088393 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.089893 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.090514 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.091411 (+ 897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.091498 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:21.091536 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 08:20:21.091621 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260808 08:20:21.091806 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:21.091946 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.086436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:44389 (local address 127.0.0.1:56976)
0808 08:20:21.087562 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.087575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:21.087668 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:21.088094 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:21.088096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:21.088111 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:21.088262 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:21.088267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.089379 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.089383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:21.090632 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:21.090645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.091533 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.091537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:21.091628 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.091654 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 08:20:21.091740 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 08:20:21.091808 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":27,"thread_start_us":120,"threads_started":1}
I20260808 08:20:21.092288 22978 tablet_service.cc:1467] Processing CreateTablet for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be (DEFAULT_TABLE table=test-table [id=237065ac0eef469284d92fe2084ad110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:21.092519 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:21.094527 23638 tablet_bootstrap.cc:492] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Bootstrap starting.
I20260808 08:20:21.094749 23640 tablet_bootstrap.cc:492] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713: Bootstrap starting.
I20260808 08:20:21.094954 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.086868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.12.13.3:44389)
0808 08:20:21.087180 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:21.087184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:21.087797 (+ 613us) server_negotiation.cc:407] Connection header received
0808 08:20:21.087869 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:21.087872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:21.087922 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:21.087997 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:21.088411 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.089242 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.090764 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.091411 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.091518 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.091576 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:21.091775 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 08:20:21.094822 (+ 3047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":81,"threads_started":1}
I20260808 08:20:21.095597 23381 tablet_service.cc:1467] Processing CreateTablet for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be (DEFAULT_TABLE table=test-table [id=237065ac0eef469284d92fe2084ad110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:21.095674 23638 tablet_bootstrap.cc:654] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:21.095814 23640 tablet_bootstrap.cc:654] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:21.095918 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:21.097012 23638 tablet_bootstrap.cc:492] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: No bootstrap required, opened a new log
I20260808 08:20:21.097126 23638 ts_tablet_manager.cc:1397] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:21.097532 23638 raft_consensus.cc:348] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.097658 23638 raft_consensus.cc:374] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:21.097697 23638 raft_consensus.cc:729] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948827c379784a05be72ff83b08d296d, State: Initialized, Role: FOLLOWER
I20260808 08:20:21.097843 23638 consensus_queue.cc:260] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.098294 23638 ts_tablet_manager.cc:1428] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 08:20:21.098745 23640 tablet_bootstrap.cc:492] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713: No bootstrap required, opened a new log
I20260808 08:20:21.098897 23640 ts_tablet_manager.cc:1397] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 08:20:21.099229 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.098965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:43401)
0808 08:20:21.099090 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.099155 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 08:20:21.098963 23648 tablet_bootstrap.cc:492] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Bootstrap starting.
I20260808 08:20:21.099442 23640 raft_consensus.cc:348] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.099555 23640 raft_consensus.cc:374] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:21.099587 23640 raft_consensus.cc:729] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307ad2ac0a53478a8deda7ad2f09a713, State: Initialized, Role: FOLLOWER
I20260808 08:20:21.099746 23640 consensus_queue.cc:260] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.100242 23640 ts_tablet_manager.cc:1428] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:21.100325 23648 tablet_bootstrap.cc:654] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:21.101830 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.101470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:45921)
0808 08:20:21.101687 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.101760 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260808 08:20:21.102101 23648 tablet_bootstrap.cc:492] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: No bootstrap required, opened a new log
I20260808 08:20:21.102190 23648 ts_tablet_manager.cc:1397] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:21.102206 23279 heartbeater.cc:503] Master 127.12.13.61:46093 was elected leader, sending a full tablet report...
I20260808 08:20:21.102675 23648 raft_consensus.cc:348] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.102798 23648 raft_consensus.cc:374] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:21.102837 23648 raft_consensus.cc:729] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930622da90e04994b43dc0f72fafdcba, State: Initialized, Role: FOLLOWER
I20260808 08:20:21.102967 23648 consensus_queue.cc:260] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.103355 23648 ts_tablet_manager.cc:1428] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:21.103488 23071 heartbeater.cc:503] Master 127.12.13.61:46093 was elected leader, sending a full tablet report...
I20260808 08:20:21.104463 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.104182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:57247)
0808 08:20:21.104332 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.104386 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 08:20:21.105458 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.105158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:43893)
0808 08:20:21.105351 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.105398 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 08:20:21.106037 23486 heartbeater.cc:503] Master 127.12.13.61:46093 was elected leader, sending a full tablet report...
W20260808 08:20:21.187592 23487 tablet.cc:2369] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:20:21.199301 23280 tablet.cc:2369] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:20:21.217914 23072 tablet.cc:2369] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:20:21.227662 23643 raft_consensus.cc:482] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:21.227877 23643 raft_consensus.cc:504] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.228366 23643 leader_election.cc:288] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235), 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:21.228892 23410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba823e8cc5654a3b8df5ad5c5cf8a6be" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930622da90e04994b43dc0f72fafdcba" is_pre_election: true
I20260808 08:20:21.228902 23005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba823e8cc5654a3b8df5ad5c5cf8a6be" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307ad2ac0a53478a8deda7ad2f09a713" is_pre_election: true
I20260808 08:20:21.229025 23410 raft_consensus.cc:2454] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 0.
I20260808 08:20:21.229089 23005 raft_consensus.cc:2454] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 0.
I20260808 08:20:21.229453 23111 leader_election.cc:302] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930622da90e04994b43dc0f72fafdcba, 948827c379784a05be72ff83b08d296d; no voters:
I20260808 08:20:21.229665 23643 raft_consensus.cc:2781] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:21.229732 23643 raft_consensus.cc:482] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:21.229794 23643 raft_consensus.cc:3037] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:21.230966 23643 raft_consensus.cc:504] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.231335 23643 leader_election.cc:288] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 election: Requested vote from peers 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235), 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389)
I20260808 08:20:21.231706 23410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba823e8cc5654a3b8df5ad5c5cf8a6be" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930622da90e04994b43dc0f72fafdcba"
I20260808 08:20:21.231716 23005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba823e8cc5654a3b8df5ad5c5cf8a6be" candidate_uuid: "948827c379784a05be72ff83b08d296d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307ad2ac0a53478a8deda7ad2f09a713"
I20260808 08:20:21.231835 23005 raft_consensus.cc:3037] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:21.231840 23410 raft_consensus.cc:3037] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:21.232762 23410 raft_consensus.cc:2454] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 1.
I20260808 08:20:21.232854 23005 raft_consensus.cc:2454] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948827c379784a05be72ff83b08d296d in term 1.
I20260808 08:20:21.233103 23111 leader_election.cc:302] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930622da90e04994b43dc0f72fafdcba, 948827c379784a05be72ff83b08d296d; no voters:
I20260808 08:20:21.233316 23643 raft_consensus.cc:2781] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:21.233512 23643 raft_consensus.cc:686] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 1 LEADER]: Becoming Leader. State: Replica: 948827c379784a05be72ff83b08d296d, State: Running, Role: LEADER
I20260808 08:20:21.233701 23643 consensus_queue.cc:237] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } }
I20260808 08:20:21.234306 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.234043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:49247)
0808 08:20:21.234169 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.234229 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 08:20:21.235894 22509 catalog_manager.cc:5495] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948827c379784a05be72ff83b08d296d (127.12.13.2). New cstate: current_term: 1 leader_uuid: "948827c379784a05be72ff83b08d296d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948827c379784a05be72ff83b08d296d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:21.249382 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.246607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.12.13.2:41789)
0808 08:20:21.246747 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:21.246750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:21.246765 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:20:21.246828 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:21.246830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:21.246893 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:21.246959 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:21.247183 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.247685 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.248490 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:21.248693 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:21.248766 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:21.249004 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:21.249070 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:21.249122 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:21.249216 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 08:20:21.249261 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:20:21.251901 23005 raft_consensus.cc:1261] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 307ad2ac0a53478a8deda7ad2f09a713 [term 1 FOLLOWER]: Refusing update from remote peer 948827c379784a05be72ff83b08d296d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:21.251940 23410 raft_consensus.cc:1261] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Refusing update from remote peer 948827c379784a05be72ff83b08d296d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:21.252501 23655 consensus_queue.cc:1035] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [LEADER]: Connected to new peer: Peer: permanent_uuid: "930622da90e04994b43dc0f72fafdcba" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:21.252714 23643 consensus_queue.cc:1035] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [LEADER]: Connected to new peer: Peer: permanent_uuid: "307ad2ac0a53478a8deda7ad2f09a713" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:21.254891 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.254527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:38121)
0808 08:20:21.254728 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.254803 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 08:20:21.259380 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.259057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:40695)
0808 08:20:21.259166 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.259234 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 08:20:21.261952 23661 mvcc.cc:205] Tried to move back new op lower bound from 7316181898242445312 to 7316181898174918656. Current Snapshot: MvccSnapshot[applied={T|T < 7316181898242445312}]
I20260808 08:20:21.262399 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.261935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.3:52851)
0808 08:20:21.262258 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.262321 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260808 08:20:21.263185 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.262569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33243 (local address 127.12.13.2:41277)
0808 08:20:21.263008 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.263074 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
I20260808 08:20:21.265190 22509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:21.265399 22509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:21.269554 22509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ff3ab74c44b4b3398655f227749a56e: Sending DeleteTablet for 3 replicas of tablet ba823e8cc5654a3b8df5ad5c5cf8a6be
I20260808 08:20:21.270339 23180 tablet_service.cc:1514] Processing DeleteTablet for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:21 UTC) from {username='oryx', principal='oryx/127.12.13.61@KRBTEST.COM'} at 127.0.0.1:47410
I20260808 08:20:21.270638 22978 tablet_service.cc:1514] Processing DeleteTablet for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:21 UTC) from {username='oryx', principal='oryx/127.12.13.61@KRBTEST.COM'} at 127.0.0.1:47540
I20260808 08:20:21.270721 23381 tablet_service.cc:1514] Processing DeleteTablet for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:21 UTC) from {username='oryx', principal='oryx/127.12.13.61@KRBTEST.COM'} at 127.0.0.1:56976
I20260808 08:20:21.271102 23675 tablet_replica.cc:330] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: stopping tablet replica
I20260808 08:20:21.271198 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 22740
I20260808 08:20:21.271245 23675 raft_consensus.cc:2229] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:21.271849 23669 tablet_replica.cc:330] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: stopping tablet replica
I20260808 08:20:21.271931 23675 raft_consensus.cc:2258] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:21.272012 23669 raft_consensus.cc:2229] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:21.272294 23669 raft_consensus.cc:2258] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:21.275012 23675 ts_tablet_manager.cc:1905] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:21.278673 23675 ts_tablet_manager.cc:1918] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:21.278769 23675 log.cc:1196] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/ba823e8cc5654a3b8df5ad5c5cf8a6be
I20260808 08:20:21.279156 23675 ts_tablet_manager.cc:1939] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 930622da90e04994b43dc0f72fafdcba: Deleting consensus metadata
I20260808 08:20:21.280162 22489 catalog_manager.cc:4841] TS 930622da90e04994b43dc0f72fafdcba (127.12.13.3:44389): tablet ba823e8cc5654a3b8df5ad5c5cf8a6be (table test-table [id=237065ac0eef469284d92fe2084ad110]) successfully deleted
I20260808 08:20:21.285741 23669 ts_tablet_manager.cc:1905] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:21.288070 23669 ts_tablet_manager.cc:1918] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:21.288152 23669 log.cc:1196] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/ba823e8cc5654a3b8df5ad5c5cf8a6be
I20260808 08:20:21.288473 23669 ts_tablet_manager.cc:1939] T ba823e8cc5654a3b8df5ad5c5cf8a6be P 948827c379784a05be72ff83b08d296d: Deleting consensus metadata
I20260808 08:20:21.289609 22489 catalog_manager.cc:4841] TS 948827c379784a05be72ff83b08d296d (127.12.13.2:41789): tablet ba823e8cc5654a3b8df5ad5c5cf8a6be (table test-table [id=237065ac0eef469284d92fe2084ad110]) successfully deleted
I20260808 08:20:21.292239 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 23084
I20260808 08:20:21.292493 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:21.292152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38235 (local address 127.0.0.1:47546)
0808 08:20:21.292288 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:21.292387 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 08:20:21.294715 22491 catalog_manager.cc:4584] TS 307ad2ac0a53478a8deda7ad2f09a713 (127.12.13.1:38235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba823e8cc5654a3b8df5ad5c5cf8a6be: Network error: Client connection negotiation failed: client connection to 127.12.13.1:38235: connect: Connection refused (error 111)
I20260808 08:20:21.308856 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 23286
I20260808 08:20:21.326277 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 22304
I20260808 08:20:21.336371 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 22471
2026-08-08T08:20:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7474 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:21 dist-test-slave-6ntk krb5kdc[23679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:21 dist-test-slave-6ntk krb5kdc[23679](info): set up 2 sockets
Aug 08 08:20:21 dist-test-slave-6ntk krb5kdc[23679](info): commencing operation
krb5kdc: starting...
W20260808 08:20:22.609287 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:22 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab.
Entry for principal oryx/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab.
I20260808 08:20:22.749459 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33841
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab
--principal=oryx/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:22.870235 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:22.870548 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:22.870639 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:22.874758 23697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:22.874823 23697 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 08:20:22.874841 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:22.874861 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:22.874876 23697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:22.874922 23697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:22.880398 23697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33841
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.62.keytab
--principal=oryx/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:22.881541 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:22.882674 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:20:22.892242 23697 server_base.cc:1034] running on GCE node
W20260808 08:20:22.892198 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:22.892213 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:22.892539 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:22.893215 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:22.894538 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:22.895712 23697 hybrid_clock.cc:648] HybridClock initialized: now 1786177222895691 us; error 36 us; skew 500 ppm
Aug 08 08:20:22 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177222, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:22.898577 23697 init.cc:377] Logged in from keytab as oryx/127.12.13.62@KRBTEST.COM (short username oryx)
I20260808 08:20:22.899457 23697 webserver.cc:460] Webserver started at http://127.12.13.62:40119/ using document root <none> and password file <none>
I20260808 08:20:22.900020 23697 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:22.900080 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:22.900256 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:22.902421 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9"
format_stamp: "Formatted at 2026-08-08 08:20:22 on dist-test-slave-6ntk"
server_key: "ef4a4d341be99836570b8472a103767c"
server_key_iv: "aefa5c5d2fbc12c967192f72a8a08e22"
server_key_version: "encryptionkey@0"
I20260808 08:20:22.902959 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9"
format_stamp: "Formatted at 2026-08-08 08:20:22 on dist-test-slave-6ntk"
server_key: "ef4a4d341be99836570b8472a103767c"
server_key_iv: "aefa5c5d2fbc12c967192f72a8a08e22"
server_key_version: "encryptionkey@0"
I20260808 08:20:22.906883 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:22.909616 23716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:22.911021 23697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:20:22.911130 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9"
format_stamp: "Formatted at 2026-08-08 08:20:22 on dist-test-slave-6ntk"
server_key: "ef4a4d341be99836570b8472a103767c"
server_key_iv: "aefa5c5d2fbc12c967192f72a8a08e22"
server_key_version: "encryptionkey@0"
I20260808 08:20:22.911223 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:22.928627 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:22.931962 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:22.932130 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:22.940528 23697 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:33841
I20260808 08:20:22.940554 23814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:33841 every 8 connection(s)
I20260808 08:20:22.941540 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:22.944603 23815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:22.947431 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 23697
I20260808 08:20:22.947573 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:22.947860 12340 external_mini_cluster.cc:1428] Setting key c560671e31c3b21c7d21ae588b295c56
I20260808 08:20:22.951133 23815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Bootstrap starting.
I20260808 08:20:22.953435 23815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:22.954283 23815 log.cc:824] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:22.956231 23815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: No bootstrap required, opened a new log
Aug 08 08:20:22 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:22.960033 23815 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33841 } }
I20260808 08:20:22.960320 23815 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:22.960417 23815 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7ea0f31cdd45e5a54b6b7b17ba3ae9, State: Initialized, Role: FOLLOWER
I20260808 08:20:22.960939 23815 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33841 } }
I20260808 08:20:22.961125 23815 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:22.961262 23815 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:22.961427 23815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:22.962410 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:22.949366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.12.13.62:33841)
0808 08:20:22.949842 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:22.949912 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:22.949965 (+ 53us) server_negotiation.cc:407] Connection header received
0808 08:20:22.950564 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:22.950580 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:22.950919 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:22.951261 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:22.952122 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:22.953110 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:22.954050 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:22.954595 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:22.954767 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:22.957321 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:22.957335 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:22.957350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:22.957377 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:22.959186 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:22.959675 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:22.959680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:22.959684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:22.959797 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:22.960103 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:22.960106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:22.960107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:22.960626 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:22.960837 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:22.961172 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 08:20:22.961546 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":152,"threads_started":1}
I20260808 08:20:22.962496 23815 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33841 } }
I20260808 08:20:22.962870 23815 leader_election.cc:302] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7ea0f31cdd45e5a54b6b7b17ba3ae9; no voters:
I20260808 08:20:22.963212 23815 leader_election.cc:288] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:22.963371 23823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:22.963644 23823 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [term 1 LEADER]: Becoming Leader. State: Replica: ff7ea0f31cdd45e5a54b6b7b17ba3ae9, State: Running, Role: LEADER
I20260808 08:20:22.963971 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33841 } }
I20260808 08:20:22.964571 23815 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:22.965157 23825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7ea0f31cdd45e5a54b6b7b17ba3ae9. Latest consensus state: current_term: 1 leader_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33841 } } }
I20260808 08:20:22.965312 23825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:22.965592 23827 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ea0f31cdd45e5a54b6b7b17ba3ae9" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33841 } } }
I20260808 08:20:22.965721 23827 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:22.965806 23832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:22.969184 23832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:22.975921 23832 catalog_manager.cc:1347] Generated new cluster ID: 94506c53712d468e908aaca14320adba
I20260808 08:20:22.976009 23832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:22.999666 23832 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:23.001077 23832 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:23.011148 23832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Generated new TSK 0
I20260808 08:20:23.012068 23832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab.
Entry for principal oryx/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab.
I20260808 08:20:23.045544 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33841
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:23.169379 23850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:23.169687 23850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:23.169781 23850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:23.173979 23850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:23.174047 23850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:23.174088 23850 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:20:23.174193 23850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:23.179610 23850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:33841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:23.180778 23850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:23.181915 23850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:23.192252 23862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:23.192246 23865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:23.192466 23850 server_base.cc:1034] running on GCE node
W20260808 08:20:23.192246 23860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:23.193001 23850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:23.193679 23850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:23.194887 23850 hybrid_clock.cc:648] HybridClock initialized: now 1786177223194872 us; error 34 us; skew 500 ppm
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:23.197715 23850 init.cc:377] Logged in from keytab as oryx/127.12.13.1@KRBTEST.COM (short username oryx)
I20260808 08:20:23.198653 23850 webserver.cc:460] Webserver started at http://127.12.13.1:35563/ using document root <none> and password file <none>
I20260808 08:20:23.199203 23850 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:23.199263 23850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:23.199448 23850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:23.201532 23850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "b60c036eb9564c9cb48834317220c923"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "065817b469445170139ddc816e2f1b02"
server_key_iv: "39253d1a15e7ef45885f56f046aca909"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.202134 23850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "b60c036eb9564c9cb48834317220c923"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "065817b469445170139ddc816e2f1b02"
server_key_iv: "39253d1a15e7ef45885f56f046aca909"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.206013 23850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:23.208493 23873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:23.209777 23850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:23.209933 23850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "b60c036eb9564c9cb48834317220c923"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "065817b469445170139ddc816e2f1b02"
server_key_iv: "39253d1a15e7ef45885f56f046aca909"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.210049 23850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:23.228255 23850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:23.231734 23850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:23.231909 23850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:23.233491 23850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:23.233546 23850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:23.233592 23850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:23.233661 23850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:23.245405 23850 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:35393
I20260808 08:20:23.245469 24064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:35393 every 8 connection(s)
I20260808 08:20:23.246438 23850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:23.248555 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.234301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40475 (local address 127.12.13.62:33841)
0808 08:20:23.234524 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.234531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.235413 (+ 882us) server_negotiation.cc:407] Connection header received
0808 08:20:23.236735 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.236742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.236816 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.236896 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.238843 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.239567 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.240746 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.240993 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.241114 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.244630 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.244637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.244639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.244674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.246419 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.247040 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.247047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.247052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.247157 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.247557 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.247559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.247560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.247743 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.247828 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.248286 (+ 458us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.248431 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 08:20:23.249397 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.234709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.12.13.1:40475)
0808 08:20:23.235225 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.235285 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.236465 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.237359 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.237370 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.237909 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:23.238657 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.238682 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.239706 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.239710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.240607 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.240616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.241170 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.241175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.241407 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.242186 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:23.242203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:23.244437 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:23.246560 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.246566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.246581 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.246919 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.247272 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.247278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.247282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.247465 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.247840 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:23.247844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:23.248072 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.248230 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.248526 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260808 08:20:23.250705 24065 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33841
I20260808 08:20:23.250964 24065 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:23.251562 24065 heartbeater.cc:511] Master 127.12.13.62:33841 requested a full tablet report, sending...
I20260808 08:20:23.253388 23740 ts_manager.cc:194] Registered new tserver with Master: b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393)
I20260808 08:20:23.254300 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 23850
I20260808 08:20:23.254457 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:23.254765 12340 external_mini_cluster.cc:1428] Setting key 2c723d9e436e7b5a39b7f6ab44053128
I20260808 08:20:23.255096 23740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:40475
I20260808 08:20:23.261538 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.253415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:45401 (local address 127.12.13.62:33841)
0808 08:20:23.253544 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.253548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.253696 (+ 148us) server_negotiation.cc:407] Connection header received
0808 08:20:23.253801 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.253803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.253841 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.253957 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.254460 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.255194 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.256351 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.256644 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.256727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.258386 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.258389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.258391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.258412 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.259885 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.260307 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.260309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.260310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.260397 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.260699 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.260705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.260709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.260884 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.260995 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.261255 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.261375 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:20:23.261543 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.253263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.12.13.1:45401)
0808 08:20:23.253600 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.253617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.253723 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.253976 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.253978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.254188 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:23.254309 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.254316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.255376 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.255379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.256183 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.256192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.256899 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.256905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.257034 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.257706 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:23.257726 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:23.258245 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:23.260005 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.260009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.260011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.260217 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.260502 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.260506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.260509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.260604 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.260985 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:23.260988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:23.261091 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.261201 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.261348 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for oryx/127.12.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab.
Entry for principal oryx/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab.
I20260808 08:20:23.289963 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33841
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:23.410853 24076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:23.411119 24076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:23.411163 24076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:23.415436 24076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:23.415505 24076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:23.415545 24076 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:20:23.415623 24076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:23.421149 24076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:33841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:23.422390 24076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:23.423480 24076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:20:23.433570 24076 server_base.cc:1034] running on GCE node
W20260808 08:20:23.433528 24081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:23.433523 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:23.433528 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:23.434295 24076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:23.434981 24076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:23.436149 24076 hybrid_clock.cc:648] HybridClock initialized: now 1786177223436133 us; error 27 us; skew 500 ppm
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:23.439065 24076 init.cc:377] Logged in from keytab as oryx/127.12.13.2@KRBTEST.COM (short username oryx)
I20260808 08:20:23.440089 24076 webserver.cc:460] Webserver started at http://127.12.13.2:46363/ using document root <none> and password file <none>
I20260808 08:20:23.440706 24076 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:23.440771 24076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:23.441051 24076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:23.443367 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "c86ad0bbaf7f432baf89a8723dba8ec2"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "db576dc60c48bf8b5372f3614fb420c0"
server_key_iv: "5380b9e2f6771fafe9efaa3a3a221a9e"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.444047 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "c86ad0bbaf7f432baf89a8723dba8ec2"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "db576dc60c48bf8b5372f3614fb420c0"
server_key_iv: "5380b9e2f6771fafe9efaa3a3a221a9e"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.448097 24076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:23.450839 24099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:23.452330 24076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:23.452486 24076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "c86ad0bbaf7f432baf89a8723dba8ec2"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "db576dc60c48bf8b5372f3614fb420c0"
server_key_iv: "5380b9e2f6771fafe9efaa3a3a221a9e"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.452616 24076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:23.474153 24076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:23.477304 24076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:23.477540 24076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:23.479259 24076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:23.479315 24076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:23.479367 24076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:23.479383 24076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:23.491132 24076 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:38645
I20260808 08:20:23.491179 24268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:38645 every 8 connection(s)
I20260808 08:20:23.492340 24076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:23.493629 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.480022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:39601 (local address 127.12.13.62:33841)
0808 08:20:23.480161 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.480165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.481263 (+ 1098us) server_negotiation.cc:407] Connection header received
0808 08:20:23.482375 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.482379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.482444 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.482562 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.484003 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.484927 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.486450 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.486680 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.486798 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.489566 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.489570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.489573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.489610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.491425 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.491918 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.491924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.491929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.492029 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.492375 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.492382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.492385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.492627 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.492741 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.493335 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.493465 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:20:23.494517 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.480619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.12.13.2:39601)
0808 08:20:23.481113 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.481158 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.482116 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.482884 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.482896 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.483272 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:23.483829 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.483843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.485077 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.485080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.486319 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.486329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.486875 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.486880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.487057 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.487699 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:23.487721 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:23.489391 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:23.491540 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.491545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.491556 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.491808 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.492127 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.492129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.492130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.492271 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.492779 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:23.492785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:23.493083 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.493288 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.493632 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
I20260808 08:20:23.495810 24269 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33841
I20260808 08:20:23.495968 24269 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:23.496559 24269 heartbeater.cc:511] Master 127.12.13.62:33841 requested a full tablet report, sending...
I20260808 08:20:23.497735 23740 ts_manager.cc:194] Registered new tserver with Master: c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645)
I20260808 08:20:23.498155 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 24076
I20260808 08:20:23.498268 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:23.498399 23740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:39601
I20260808 08:20:23.499230 12340 external_mini_cluster.cc:1428] Setting key f17d47ec266295a17958d94b659e0aea
I20260808 08:20:23.507727 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.500009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:44945 (local address 127.12.13.62:33841)
0808 08:20:23.500141 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.500146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.500351 (+ 205us) server_negotiation.cc:407] Connection header received
0808 08:20:23.500499 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.500502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.500553 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.500641 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.501160 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.501741 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.502626 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.502863 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.502977 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.504342 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.504346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.504348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.504375 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.505943 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.506478 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.506480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.506482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.506558 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.506871 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.506872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.506874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.507076 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.507154 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.507503 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.507603 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:20:23.507926 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.499976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.12.13.2:44945)
0808 08:20:23.500252 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.500268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.500385 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.500674 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.500677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.500909 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:23.501021 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.501028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.501926 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.501929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.502505 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.502512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.502981 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.502985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.503077 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.503653 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:23.503668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:23.504027 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:23.506067 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.506073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.506078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.506373 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.506652 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.506666 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.506670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.506786 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.507182 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:23.507186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:23.507308 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.507437 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.507617 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab.
Entry for principal oryx/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab.
I20260808 08:20:23.534279 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33841
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:23.653033 24286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:23.653327 24286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:23.653411 24286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:23.657641 24286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:23.657711 24286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:23.657783 24286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 08:20:23.657886 24286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:23.663475 24286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.13.62:33841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:23.664737 24286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:23.665949 24286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:23.676458 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:23.676607 24286 server_base.cc:1034] running on GCE node
W20260808 08:20:23.676442 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:23.676440 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:23.677232 24286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:23.677959 24286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:23.679142 24286 hybrid_clock.cc:648] HybridClock initialized: now 1786177223679110 us; error 48 us; skew 500 ppm
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:23.681942 24286 init.cc:377] Logged in from keytab as oryx/127.12.13.3@KRBTEST.COM (short username oryx)
I20260808 08:20:23.682835 24286 webserver.cc:460] Webserver started at http://127.12.13.3:41761/ using document root <none> and password file <none>
I20260808 08:20:23.683390 24286 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:23.683447 24286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:23.683616 24286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:23.685727 24286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "19611cbe97ad42738e985b084fa8a1c8"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "507a862cc4d86f0a9da6ca6943fc9682"
server_key_iv: "e72ed7279c86a9643a43e7f52b4506ec"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.686322 24286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "19611cbe97ad42738e985b084fa8a1c8"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "507a862cc4d86f0a9da6ca6943fc9682"
server_key_iv: "e72ed7279c86a9643a43e7f52b4506ec"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.690214 24286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 08:20:23.692727 24310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:23.694052 24286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:23.694173 24286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "19611cbe97ad42738e985b084fa8a1c8"
format_stamp: "Formatted at 2026-08-08 08:20:23 on dist-test-slave-6ntk"
server_key: "507a862cc4d86f0a9da6ca6943fc9682"
server_key_iv: "e72ed7279c86a9643a43e7f52b4506ec"
server_key_version: "encryptionkey@0"
I20260808 08:20:23.694280 24286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:23.726365 24286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:23.729524 24286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:23.729736 24286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:23.731278 24286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:23.731328 24286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:23.731380 24286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:23.731436 24286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:23.745020 24286 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:45509
I20260808 08:20:23.745071 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:45509 every 8 connection(s)
I20260808 08:20:23.746273 24286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:23.747825 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.732346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:46033 (local address 127.12.13.62:33841)
0808 08:20:23.732512 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.732519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.733395 (+ 876us) server_negotiation.cc:407] Connection header received
0808 08:20:23.734650 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.734653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.734719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.734828 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.736313 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.737017 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.738597 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.738913 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.739053 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.743344 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.743348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.743351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.743388 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.745353 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.746040 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.746043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.746045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.746143 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.746535 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.746538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.746539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.746823 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.746929 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.747582 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.747698 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:20:23.748652 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.732746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.12.13.3:46033)
0808 08:20:23.733222 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.733274 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.734411 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.735023 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.735030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.735481 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:23.736161 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.736177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.737162 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.737168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.738452 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.738466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.739099 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.739103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.739301 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.740005 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:23.740033 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:23.743077 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:23.745479 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.745485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.745500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.745911 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.746250 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.746254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.746256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.746441 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.746984 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:23.746991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:23.747298 (+ 307us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.747458 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.747753 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
I20260808 08:20:23.749912 24484 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33841
I20260808 08:20:23.750094 24484 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:23.750635 24484 heartbeater.cc:511] Master 127.12.13.62:33841 requested a full tablet report, sending...
I20260808 08:20:23.751814 23738 ts_manager.cc:194] Registered new tserver with Master: 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:23.752493 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 24286
I20260808 08:20:23.752660 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:23.752666 23738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:46033
I20260808 08:20:23.753046 12340 external_mini_cluster.cc:1428] Setting key 7a50ac06eef24520b78ce04369d6bca8
I20260808 08:20:23.756134 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:20:23.761441 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.753459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:43207 (local address 127.12.13.62:33841)
0808 08:20:23.753597 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.753602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.753617 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:20:23.753655 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.753658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.753699 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.753787 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.754429 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.755100 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.756291 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.756598 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.756729 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.758064 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.758070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.758076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.758108 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.759712 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.760234 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.760236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.760237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.760310 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.760642 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.760644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.760645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.760811 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.760898 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.761224 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.761321 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 08:20:23.761672 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.752387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.12.13.3:43207)
0808 08:20:23.752692 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.752711 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.752821 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.753949 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.753955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.754153 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:23.754301 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.754309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.755230 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.755233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.756121 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.756130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.756748 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.756752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.756849 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.757503 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:23.757516 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:23.757928 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:23.759836 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.759841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.759844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.760125 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.760426 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:23.760431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:23.760436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:23.760547 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:23.760923 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:23.760928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:23.761052 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.761172 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.761362 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 08:20:23.784642 23819 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:23.777633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.12.13.62:33841)
0808 08:20:23.777768 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.777772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.777783 (+ 11us) server_negotiation.cc:407] Connection header received
0808 08:20:23.777825 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.777828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.777934 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.778026 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.778819 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.779820 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.780754 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.780990 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.781095 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.784482 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58746: BlockingRecv error: recv got EOF from 127.0.0.1:58746 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 08:20:23 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:23.794039 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.786189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.12.13.62:33841)
0808 08:20:23.786325 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.786329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.786448 (+ 119us) server_negotiation.cc:407] Connection header received
0808 08:20:23.786607 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.786610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.786661 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.786742 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:23.787342 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.787858 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.788651 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.788879 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.788983 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.791314 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:23.791317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:23.791319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:23.791343 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:23.792595 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.792986 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.792988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.792989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.793061 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:23.793289 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:23.793291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:23.793292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:23.793469 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:23.793572 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.793773 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.793864 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:20:23.796947 23738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:23.800451 23738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:23.815788 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.809341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.12.13.3:45509)
0808 08:20:23.810524 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.810536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.810559 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:20:23.810632 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.810637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.810827 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.811294 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:23.811447 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.812232 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.814312 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.815120 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.815236 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.815428 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.815575 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.815662 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":140,"threads_started":1}
I20260808 08:20:23.815845 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.809133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:45509 (local address 127.0.0.1:57236)
0808 08:20:23.809513 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.809573 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.809712 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.811002 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.811005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.811025 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:23.811151 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.811162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.812356 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.812361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.814164 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.814177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.815250 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.815254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.815369 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.815416 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.815538 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.815661 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":184,"thread_start_us":95,"threads_started":1}
I20260808 08:20:23.816448 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.809858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:38645 (local address 127.0.0.1:50798)
0808 08:20:23.810724 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.810738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.810818 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.811383 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.811386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.811403 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:23.811527 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.811545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.813818 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.813822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.814843 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.814856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.815907 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.815912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.816139 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.816163 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.816246 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.816328 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":93,"threads_started":1}
I20260808 08:20:23.816561 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.810392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.12.13.2:38645)
0808 08:20:23.810792 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.810804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.810826 (+ 22us) server_negotiation.cc:407] Connection header received
0808 08:20:23.810923 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.810930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.811095 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.811246 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:23.811673 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.812710 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.814974 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.815769 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.815888 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.816129 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.816300 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.816406 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":106,"threads_started":1}
I20260808 08:20:23.817164 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.809541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:35393 (local address 127.0.0.1:39724)
0808 08:20:23.810142 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.810156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.810242 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.810950 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.810956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.810979 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:23.811106 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.811111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.812408 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.812411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.814154 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.814165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.816345 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.816349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.816445 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.816469 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.816965 (+ 496us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.817042 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":182,"thread_start_us":113,"threads_started":1}
I20260808 08:20:23.817217 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.810003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.12.13.1:35393)
0808 08:20:23.810317 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.810325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.810369 (+ 44us) server_negotiation.cc:407] Connection header received
0808 08:20:23.810457 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.810463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.810692 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.810850 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:23.811218 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.812229 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.814282 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.815120 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.815307 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.815428 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.816993 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.817089 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260808 08:20:23.819617 24172 tablet_service.cc:1467] Processing CreateTablet for tablet 1533f10ebef04b409b7eb2876722b5da (DEFAULT_TABLE table=test-table [id=16febe4af0f04b0e838abcfa8e74bca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:23.819629 24384 tablet_service.cc:1467] Processing CreateTablet for tablet 1533f10ebef04b409b7eb2876722b5da (DEFAULT_TABLE table=test-table [id=16febe4af0f04b0e838abcfa8e74bca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:23.820003 23967 tablet_service.cc:1467] Processing CreateTablet for tablet 1533f10ebef04b409b7eb2876722b5da (DEFAULT_TABLE table=test-table [id=16febe4af0f04b0e838abcfa8e74bca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:23.820691 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1533f10ebef04b409b7eb2876722b5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:23.820704 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1533f10ebef04b409b7eb2876722b5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:23.820896 23967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1533f10ebef04b409b7eb2876722b5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:23.828127 24523 tablet_bootstrap.cc:492] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Bootstrap starting.
I20260808 08:20:23.828779 24525 tablet_bootstrap.cc:492] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Bootstrap starting.
I20260808 08:20:23.830219 24524 tablet_bootstrap.cc:492] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Bootstrap starting.
I20260808 08:20:23.830436 24523 tablet_bootstrap.cc:654] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:23.830920 24525 tablet_bootstrap.cc:654] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:23.831243 24523 log.cc:824] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:23.831734 24525 log.cc:824] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:23.832268 24524 tablet_bootstrap.cc:654] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:23.833073 24524 log.cc:824] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:23.833455 24523 tablet_bootstrap.cc:492] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: No bootstrap required, opened a new log
I20260808 08:20:23.833674 24523 ts_tablet_manager.cc:1397] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:20:23.833706 24525 tablet_bootstrap.cc:492] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: No bootstrap required, opened a new log
I20260808 08:20:23.833930 24525 ts_tablet_manager.cc:1397] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:20:23.835192 24524 tablet_bootstrap.cc:492] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: No bootstrap required, opened a new log
I20260808 08:20:23.835453 24524 ts_tablet_manager.cc:1397] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:20:23.836678 24523 raft_consensus.cc:348] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.836941 24523 raft_consensus.cc:374] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:23.837000 24523 raft_consensus.cc:729] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Initialized, Role: FOLLOWER
I20260808 08:20:23.837540 24523 consensus_queue.cc:260] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.838281 24269 heartbeater.cc:503] Master 127.12.13.62:33841 was elected leader, sending a full tablet report...
I20260808 08:20:23.838599 24523 ts_tablet_manager.cc:1428] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 08:20:23.839139 24525 raft_consensus.cc:348] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.839373 24525 raft_consensus.cc:374] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:23.839437 24525 raft_consensus.cc:729] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60c036eb9564c9cb48834317220c923, State: Initialized, Role: FOLLOWER
I20260808 08:20:23.839370 24524 raft_consensus.cc:348] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.839612 24524 raft_consensus.cc:374] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:23.839651 24524 raft_consensus.cc:729] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19611cbe97ad42738e985b084fa8a1c8, State: Initialized, Role: FOLLOWER
I20260808 08:20:23.839917 24525 consensus_queue.cc:260] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.840077 24524 consensus_queue.cc:260] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.840565 24065 heartbeater.cc:503] Master 127.12.13.62:33841 was elected leader, sending a full tablet report...
I20260808 08:20:23.840895 24525 ts_tablet_manager.cc:1428] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 08:20:23.840993 24524 ts_tablet_manager.cc:1428] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:20:23.841282 24484 heartbeater.cc:503] Master 127.12.13.62:33841 was elected leader, sending a full tablet report...
I20260808 08:20:23.844673 24532 raft_consensus.cc:482] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:23.844955 24532 raft_consensus.cc:504] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.846014 24532 leader_election.cc:288] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509), b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393)
I20260808 08:20:23.849826 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.846220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:35393 (local address 127.12.13.2:50653)
0808 08:20:23.846336 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.846353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.846443 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.846709 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.846712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.846731 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:23.846875 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.846882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.847766 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.847771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.848698 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.848707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.849389 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.849393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.849495 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.849517 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.849616 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.849691 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 08:20:23.849826 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.846300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:50653 (local address 127.12.13.1:35393)
0808 08:20:23.846410 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.846415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.846430 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:20:23.846540 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.846542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.846576 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.846682 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:23.846999 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.847661 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.848819 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.849244 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.849389 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.849454 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.849630 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.849701 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 08:20:23.850474 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1533f10ebef04b409b7eb2876722b5da" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923" is_pre_election: true
I20260808 08:20:23.850809 23995 raft_consensus.cc:2454] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:23.850909 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.846292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:60559 (local address 127.12.13.3:45509)
0808 08:20:23.846410 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.846414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.846875 (+ 461us) server_negotiation.cc:407] Connection header received
0808 08:20:23.846953 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.846958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.847006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.847076 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:23.847393 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.848127 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.849706 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.850360 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.850468 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.850529 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.850723 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.850793 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 08:20:23.850932 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.846220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:45509 (local address 127.12.13.2:60559)
0808 08:20:23.846776 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:23.846790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:23.846882 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:23.847105 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:23.847108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:23.847124 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:23.847269 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.847276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.848272 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.848275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.849566 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:23.849577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.850498 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.850501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:23.850603 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.850628 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 08:20:23.850708 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 08:20:23.850778 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":198,"thread_start_us":129,"threads_started":1}
I20260808 08:20:23.851402 24101 leader_election.cc:302] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b60c036eb9564c9cb48834317220c923, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:23.851593 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1533f10ebef04b409b7eb2876722b5da" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8" is_pre_election: true
I20260808 08:20:23.851720 24532 raft_consensus.cc:2781] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:23.851820 24532 raft_consensus.cc:482] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:23.851886 24532 raft_consensus.cc:3037] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:23.851871 24419 raft_consensus.cc:2454] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:23.853240 24532 raft_consensus.cc:504] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.853600 24532 leader_election.cc:288] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Requested vote from peers 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509), b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393)
I20260808 08:20:23.854028 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1533f10ebef04b409b7eb2876722b5da" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923"
I20260808 08:20:23.854028 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1533f10ebef04b409b7eb2876722b5da" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8"
I20260808 08:20:23.854156 24419 raft_consensus.cc:3037] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:23.854156 23995 raft_consensus.cc:3037] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:23.855475 23995 raft_consensus.cc:2454] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:23.855475 24419 raft_consensus.cc:2454] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:23.855795 24104 leader_election.cc:302] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, b60c036eb9564c9cb48834317220c923, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:23.856035 24532 raft_consensus.cc:2781] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:23.856243 24532 raft_consensus.cc:686] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Becoming Leader. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Running, Role: LEADER
I20260808 08:20:23.856628 24532 consensus_queue.cc:237] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } }
I20260808 08:20:23.860257 23740 catalog_manager.cc:5495] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:23.872293 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:23.868326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.12.13.2:38645)
0808 08:20:23.868493 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:23.868497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:23.868514 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:20:23.868705 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:23.868708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:23.868786 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:23.868893 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:23.869237 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.869832 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.870763 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:23.870989 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:23.871073 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:23.871339 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:23.871831 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:23.871922 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:23.872026 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 08:20:23.872081 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:20:23.878710 23995 raft_consensus.cc:1261] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:23.879102 24419 raft_consensus.cc:1261] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:23.879549 24538 consensus_queue.cc:1035] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:23.879726 24532 consensus_queue.cc:1035] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:23.886639 24544 mvcc.cc:205] Tried to move back new op lower bound from 7316181908998381568 to 7316181908919787520. Current Snapshot: MvccSnapshot[applied={T|T < 7316181908998381568}]
I20260808 08:20:23.889617 24546 mvcc.cc:205] Tried to move back new op lower bound from 7316181908998381568 to 7316181908919787520. Current Snapshot: MvccSnapshot[applied={T|T < 7316181908998381568}]
I20260808 08:20:23.889744 24545 mvcc.cc:205] Tried to move back new op lower bound from 7316181908998381568 to 7316181908919787520. Current Snapshot: MvccSnapshot[applied={T|T < 7316181908998381568}]
I20260808 08:20:23.895773 23740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:23.895946 23740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:23.898644 23740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Sending DeleteTablet for 3 replicas of tablet 1533f10ebef04b409b7eb2876722b5da
I20260808 08:20:23.899600 24172 tablet_service.cc:1514] Processing DeleteTablet for tablet 1533f10ebef04b409b7eb2876722b5da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:23 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50798
I20260808 08:20:23.899610 24384 tablet_service.cc:1514] Processing DeleteTablet for tablet 1533f10ebef04b409b7eb2876722b5da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:23 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:57236
I20260808 08:20:23.900158 23967 tablet_service.cc:1514] Processing DeleteTablet for tablet 1533f10ebef04b409b7eb2876722b5da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:23 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:39724
I20260808 08:20:23.900429 23740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 08:20:23.900758 24556 tablet_replica.cc:330] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: stopping tablet replica
W20260808 08:20:23.900859 23740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:23.900904 24557 tablet_replica.cc:330] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: stopping tablet replica
I20260808 08:20:23.901115 24556 raft_consensus.cc:2229] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:23.901367 24557 raft_consensus.cc:2229] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:23.901578 24556 raft_consensus.cc:2258] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:23.901780 24557 raft_consensus.cc:2258] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:23.902429 24555 tablet_replica.cc:330] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: stopping tablet replica
I20260808 08:20:23.903656 24557 ts_tablet_manager.cc:1905] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:23.902717 24555 raft_consensus.cc:2229] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:23.904927 24555 raft_consensus.cc:2258] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:23.906077 23967 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae4ad382b0648f8908dd721efa0de24 (DEFAULT_TABLE table=test-table [id=2c8f49457e6c44cfae57bfac208bb547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:23.906371 23967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae4ad382b0648f8908dd721efa0de24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:23.906679 24384 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae4ad382b0648f8908dd721efa0de24 (DEFAULT_TABLE table=test-table [id=2c8f49457e6c44cfae57bfac208bb547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:23.906944 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae4ad382b0648f8908dd721efa0de24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:23.908314 24525 tablet_bootstrap.cc:492] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Bootstrap starting.
I20260808 08:20:23.908545 24172 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae4ad382b0648f8908dd721efa0de24 (DEFAULT_TABLE table=test-table [id=2c8f49457e6c44cfae57bfac208bb547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:23.908824 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae4ad382b0648f8908dd721efa0de24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:23.909286 24524 tablet_bootstrap.cc:492] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Bootstrap starting.
I20260808 08:20:23.909332 24525 tablet_bootstrap.cc:654] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:23.910727 24524 tablet_bootstrap.cc:654] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:23.911214 24523 tablet_bootstrap.cc:492] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Bootstrap starting.
I20260808 08:20:23.911837 24555 ts_tablet_manager.cc:1905] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:23.912236 24523 tablet_bootstrap.cc:654] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:23.912703 24556 ts_tablet_manager.cc:1905] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:23.914314 24555 ts_tablet_manager.cc:1918] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:23.914422 24555 log.cc:1196] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/1533f10ebef04b409b7eb2876722b5da
I20260808 08:20:23.914745 24555 ts_tablet_manager.cc:1939] T 1533f10ebef04b409b7eb2876722b5da P 19611cbe97ad42738e985b084fa8a1c8: Deleting consensus metadata
I20260808 08:20:23.915516 24556 ts_tablet_manager.cc:1918] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:23.916057 23722 catalog_manager.cc:4841] TS 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509): tablet 1533f10ebef04b409b7eb2876722b5da (table test-table [id=16febe4af0f04b0e838abcfa8e74bca2]) successfully deleted
I20260808 08:20:23.916563 24524 tablet_bootstrap.cc:492] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: No bootstrap required, opened a new log
I20260808 08:20:23.916632 24524 ts_tablet_manager.cc:1397] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260808 08:20:23.916898 24557 ts_tablet_manager.cc:1918] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:23.917011 24557 log.cc:1196] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/1533f10ebef04b409b7eb2876722b5da
I20260808 08:20:23.917057 24524 raft_consensus.cc:348] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:23.917156 24524 raft_consensus.cc:374] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:23.917186 24524 raft_consensus.cc:729] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19611cbe97ad42738e985b084fa8a1c8, State: Initialized, Role: FOLLOWER
I20260808 08:20:23.917376 24557 ts_tablet_manager.cc:1939] T 1533f10ebef04b409b7eb2876722b5da P b60c036eb9564c9cb48834317220c923: Deleting consensus metadata
I20260808 08:20:23.917320 24524 consensus_queue.cc:260] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:23.917613 24524 ts_tablet_manager.cc:1428] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:23.918674 24556 log.cc:1196] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/1533f10ebef04b409b7eb2876722b5da
I20260808 08:20:23.918776 23718 catalog_manager.cc:4841] TS b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393): tablet 1533f10ebef04b409b7eb2876722b5da (table test-table [id=16febe4af0f04b0e838abcfa8e74bca2]) successfully deleted
I20260808 08:20:23.919060 24556 ts_tablet_manager.cc:1939] T 1533f10ebef04b409b7eb2876722b5da P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting consensus metadata
I20260808 08:20:23.920265 23722 catalog_manager.cc:4841] TS c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645): tablet 1533f10ebef04b409b7eb2876722b5da (table test-table [id=16febe4af0f04b0e838abcfa8e74bca2]) successfully deleted
I20260808 08:20:23.920959 24523 tablet_bootstrap.cc:492] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: No bootstrap required, opened a new log
I20260808 08:20:23.921052 24523 ts_tablet_manager.cc:1397] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260808 08:20:23.921268 24525 tablet_bootstrap.cc:492] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: No bootstrap required, opened a new log
I20260808 08:20:23.921386 24525 ts_tablet_manager.cc:1397] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260808 08:20:23.921501 24523 raft_consensus.cc:348] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:23.921623 24523 raft_consensus.cc:374] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:23.921655 24523 raft_consensus.cc:729] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Initialized, Role: FOLLOWER
I20260808 08:20:23.921774 24523 consensus_queue.cc:260] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:23.922060 24523 ts_tablet_manager.cc:1428] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:23.922286 24525 raft_consensus.cc:348] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:23.922430 24525 raft_consensus.cc:374] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:23.922487 24525 raft_consensus.cc:729] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60c036eb9564c9cb48834317220c923, State: Initialized, Role: FOLLOWER
I20260808 08:20:23.922662 24525 consensus_queue.cc:260] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:23.922960 24525 ts_tablet_manager.cc:1428] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260808 08:20:23.993561 24270 tablet.cc:2369] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:20:23.997380 24486 tablet.cc:2369] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:20:23.998473 24066 tablet.cc:2369] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:20:24.003795 24538 raft_consensus.cc:482] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:24.003949 24538 raft_consensus.cc:504] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.004446 24538 leader_election.cc:288] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.004971 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae4ad382b0648f8908dd721efa0de24" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8" is_pre_election: true
I20260808 08:20:24.004985 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae4ad382b0648f8908dd721efa0de24" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923" is_pre_election: true
I20260808 08:20:24.005143 24419 raft_consensus.cc:2454] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:24.005177 23995 raft_consensus.cc:2454] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:24.005504 24104 leader_election.cc:302] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.005697 24538 raft_consensus.cc:2781] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:24.005781 24538 raft_consensus.cc:482] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:24.005847 24538 raft_consensus.cc:3037] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.006953 24538 raft_consensus.cc:504] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.007354 24538 leader_election.cc:288] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Requested vote from peers b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.007736 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae4ad382b0648f8908dd721efa0de24" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8"
I20260808 08:20:24.007730 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae4ad382b0648f8908dd721efa0de24" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923"
I20260808 08:20:24.007876 23995 raft_consensus.cc:3037] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.007876 24419 raft_consensus.cc:3037] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.008899 24419 raft_consensus.cc:2454] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:24.009143 23995 raft_consensus.cc:2454] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:24.009186 24104 leader_election.cc:302] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.009374 24538 raft_consensus.cc:2781] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:24.009466 24538 raft_consensus.cc:686] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Becoming Leader. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Running, Role: LEADER
I20260808 08:20:24.009644 24538 consensus_queue.cc:237] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.011526 23740 catalog_manager.cc:5495] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } health_report { overall_health: UNKNOWN } } }
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177222, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.62@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:24.084201 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.075898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33841 (local address 127.0.0.1:58764)
0808 08:20:24.076237 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.076255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:24.076386 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:24.076643 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:24.076645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:24.077076 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:24.077178 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.077185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.077995 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.077998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.078554 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.078562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.078972 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.078975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.079035 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.079593 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:24.079612 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:24.081057 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:24.082710 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:24.082719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:24.082725 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:24.082966 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:24.083207 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:24.083209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:24.083211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:24.083354 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:24.083697 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:24.083702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:24.083811 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 08:20:24.083907 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 08:20:24.084036 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":150,"threads_started":1}
I20260808 08:20:24.084362 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.075904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.12.13.62:33841)
0808 08:20:24.076062 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.076066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.076345 (+ 279us) server_negotiation.cc:407] Connection header received
0808 08:20:24.076471 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.076473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.076518 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.076604 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:24.077303 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.077825 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.078657 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.078870 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.078943 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.081191 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:24.081194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:24.081196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:24.081218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:24.082611 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:24.083054 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:24.083056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:24.083057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:24.083127 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:24.083437 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:24.083439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:24.083440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:24.083607 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:24.083680 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.083963 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.084172 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":72}
I20260808 08:20:24.088762 23740 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:58764:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 08:20:24.089483 23740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:24.094839 23967 tablet_service.cc:1467] Processing CreateTablet for tablet 3c27002777c14b03b27d145bedc170a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0347465ffa5f41158f19341dff92b0b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:20:24.094844 24172 tablet_service.cc:1467] Processing CreateTablet for tablet 3c27002777c14b03b27d145bedc170a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0347465ffa5f41158f19341dff92b0b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:20:24.094856 24384 tablet_service.cc:1467] Processing CreateTablet for tablet 3c27002777c14b03b27d145bedc170a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0347465ffa5f41158f19341dff92b0b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 08:20:24.095289 23967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c27002777c14b03b27d145bedc170a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.095284 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c27002777c14b03b27d145bedc170a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.095284 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c27002777c14b03b27d145bedc170a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.098233 24524 tablet_bootstrap.cc:492] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: Bootstrap starting.
I20260808 08:20:24.098232 24523 tablet_bootstrap.cc:492] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: Bootstrap starting.
I20260808 08:20:24.098274 24525 tablet_bootstrap.cc:492] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: Bootstrap starting.
I20260808 08:20:24.099244 24525 tablet_bootstrap.cc:654] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.099244 24523 tablet_bootstrap.cc:654] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.099304 24524 tablet_bootstrap.cc:654] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.100566 24523 tablet_bootstrap.cc:492] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: No bootstrap required, opened a new log
I20260808 08:20:24.100625 24524 tablet_bootstrap.cc:492] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: No bootstrap required, opened a new log
I20260808 08:20:24.100625 24525 tablet_bootstrap.cc:492] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: No bootstrap required, opened a new log
I20260808 08:20:24.100687 24523 ts_tablet_manager.cc:1397] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 08:20:24.100704 24524 ts_tablet_manager.cc:1397] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:24.100704 24525 ts_tablet_manager.cc:1397] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:24.101084 24523 raft_consensus.cc:348] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.101205 24523 raft_consensus.cc:374] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.101161 24525 raft_consensus.cc:348] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.101161 24524 raft_consensus.cc:348] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.101245 24523 raft_consensus.cc:729] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.101271 24525 raft_consensus.cc:374] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.101271 24524 raft_consensus.cc:374] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.101308 24525 raft_consensus.cc:729] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60c036eb9564c9cb48834317220c923, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.101308 24524 raft_consensus.cc:729] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19611cbe97ad42738e985b084fa8a1c8, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.101388 24523 consensus_queue.cc:260] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.101438 24524 consensus_queue.cc:260] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.101438 24525 consensus_queue.cc:260] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.101682 24523 ts_tablet_manager.cc:1428] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:24.101706 24524 ts_tablet_manager.cc:1428] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:24.101732 24525 ts_tablet_manager.cc:1428] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:24.101940 24538 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.102000 24533 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.102087 24538 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.102144 24533 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.102519 24534 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.102689 24534 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.287822 24533 raft_consensus.cc:482] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:24.287967 24533 raft_consensus.cc:504] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.289139 24533 leader_election.cc:288] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.293496 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.290036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:57099 (local address 127.12.13.3:45509)
0808 08:20:24.290189 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.290194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.290280 (+ 86us) server_negotiation.cc:407] Connection header received
0808 08:20:24.290509 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.290512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.290560 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.290654 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:24.290954 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.291483 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.292526 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.292936 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.293043 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.293105 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.293285 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.293364 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:20:24.293522 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.289798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:45509 (local address 127.12.13.1:57099)
0808 08:20:24.290184 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.290201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:24.290364 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:24.290701 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:24.290704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:24.290724 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:24.290821 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.290827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.291620 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.291622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.292368 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.292407 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.293075 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.293084 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.293165 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.293183 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 08:20:24.293242 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 08:20:24.293320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260808 08:20:24.293540 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.289479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:49027 (local address 127.12.13.2:38645)
0808 08:20:24.289623 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.289627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.289825 (+ 198us) server_negotiation.cc:407] Connection header received
0808 08:20:24.289947 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.289954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.290022 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.290136 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:24.290701 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.291304 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.292455 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.292867 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.292954 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.293041 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.293359 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.293435 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:20:24.293532 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.289348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:38645 (local address 127.12.13.1:49027)
0808 08:20:24.289657 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.289675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:24.289803 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:24.290262 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:24.290265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:24.290290 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:24.290558 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.290566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.291416 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.291420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.292314 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.292324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.292994 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.292999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.293084 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.293157 (+ 73us) client_negotiation.cc:769] Sending connection context
0808 08:20:24.293312 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 08:20:24.293378 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":27,"thread_start_us":134,"threads_started":1}
I20260808 08:20:24.294196 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c27002777c14b03b27d145bedc170a7" candidate_uuid: "b60c036eb9564c9cb48834317220c923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8" is_pre_election: true
I20260808 08:20:24.294252 24200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c27002777c14b03b27d145bedc170a7" candidate_uuid: "b60c036eb9564c9cb48834317220c923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" is_pre_election: true
I20260808 08:20:24.294328 24419 raft_consensus.cc:2454] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60c036eb9564c9cb48834317220c923 in term 0.
I20260808 08:20:24.294559 24200 raft_consensus.cc:2454] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60c036eb9564c9cb48834317220c923 in term 0.
I20260808 08:20:24.294750 23877 leader_election.cc:302] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, b60c036eb9564c9cb48834317220c923; no voters:
I20260808 08:20:24.295027 24533 raft_consensus.cc:2781] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:24.295104 24533 raft_consensus.cc:482] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:24.295158 24533 raft_consensus.cc:3037] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.296639 24533 raft_consensus.cc:504] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.296988 24533 leader_election.cc:288] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [CANDIDATE]: Term 1 election: Requested vote from peers c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.297403 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c27002777c14b03b27d145bedc170a7" candidate_uuid: "b60c036eb9564c9cb48834317220c923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8"
I20260808 08:20:24.297420 24200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c27002777c14b03b27d145bedc170a7" candidate_uuid: "b60c036eb9564c9cb48834317220c923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2"
I20260808 08:20:24.297531 24419 raft_consensus.cc:3037] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.297531 24200 raft_consensus.cc:3037] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.298699 24200 raft_consensus.cc:2454] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60c036eb9564c9cb48834317220c923 in term 1.
I20260808 08:20:24.298699 24419 raft_consensus.cc:2454] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60c036eb9564c9cb48834317220c923 in term 1.
I20260808 08:20:24.298972 23877 leader_election.cc:302] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b60c036eb9564c9cb48834317220c923, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.299149 24533 raft_consensus.cc:2781] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:24.299377 24533 raft_consensus.cc:686] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [term 1 LEADER]: Becoming Leader. State: Replica: b60c036eb9564c9cb48834317220c923, State: Running, Role: LEADER
I20260808 08:20:24.299664 24533 consensus_queue.cc:237] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.300899 24586 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: TxnStatusTablet state changed. Reason: New leader b60c036eb9564c9cb48834317220c923. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.301033 24586 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:20:24.301374 24589 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 08:20:24.301537 24589 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 08:20:24.302289 23738 catalog_manager.cc:5495] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 reported cstate change: term changed from 0 to 1, leader changed from <none> to b60c036eb9564c9cb48834317220c923 (127.12.13.1). New cstate: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:24.344764 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.339609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.12.13.1:35393)
0808 08:20:24.339736 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.339741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.339757 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:20:24.339932 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.339935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.340012 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.340133 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:24.340376 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.341182 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.342071 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.342422 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.342533 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.342794 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.344251 (+ 1457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.344379 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.344545 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.344611 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:20:24.345281 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.339470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:35393 (local address 127.0.0.1:39726)
0808 08:20:24.339635 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.339653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:24.339766 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:24.340153 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:24.340155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:24.340164 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:20:24.340270 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.340274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.341294 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.341296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.341955 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.341965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.342536 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.342539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.342598 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.342647 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.344381 (+ 1734us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.344400 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 08:20:24.344534 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 08:20:24.344714 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 08:20:24.349001 24419 raft_consensus.cc:1261] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Refusing update from remote peer b60c036eb9564c9cb48834317220c923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.349615 24586 consensus_queue.cc:1035] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.352324 24534 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: TxnStatusTablet state changed. Reason: New leader b60c036eb9564c9cb48834317220c923. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.352445 24534 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.354704 24586 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.354823 24586 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:20:24.354815 24200 raft_consensus.cc:1261] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Refusing update from remote peer b60c036eb9564c9cb48834317220c923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.355624 24533 consensus_queue.cc:1035] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.355963 24534 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.356081 24534 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P 19611cbe97ad42738e985b084fa8a1c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.358556 24538 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: TxnStatusTablet state changed. Reason: New leader b60c036eb9564c9cb48834317220c923. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.358685 24538 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.358793 24538 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.358879 24538 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P c86ad0bbaf7f432baf89a8723dba8ec2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 08:20:24.371129 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.367316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:35393 (local address 127.12.13.2:44651)
0808 08:20:24.367615 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.367629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:24.367722 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:24.368015 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:24.368019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:24.368027 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:20:24.368160 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.368165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.369190 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.369193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.369815 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.369823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.370442 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.370445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.370508 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.370555 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.370878 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.370882 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 08:20:24.370929 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 08:20:24.370999 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20260808 08:20:24.371135 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.367427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:44651 (local address 127.12.13.1:35393)
0808 08:20:24.367546 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.367551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.367703 (+ 152us) server_negotiation.cc:407] Connection header received
0808 08:20:24.367810 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.367813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.367876 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.367963 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:24.368270 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.369076 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.369982 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.370323 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.370430 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.370687 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.370774 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.370852 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.370943 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.370998 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 08:20:24.373584 24586 tablet_replica.cc:439] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b60c036eb9564c9cb48834317220c923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } } }
I20260808 08:20:24.373721 24586 tablet_replica.cc:442] T 3c27002777c14b03b27d145bedc170a7 P b60c036eb9564c9cb48834317220c923: This TxnStatusTablet replica's current role is: LEADER
I20260808 08:20:24.378734 23995 raft_consensus.cc:1261] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.378765 24419 raft_consensus.cc:1261] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.379184 24538 consensus_queue.cc:1035] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.379343 24532 consensus_queue.cc:1035] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.403919 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.399751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:54513 (local address 127.12.13.2:38645)
0808 08:20:24.399874 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.399881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.400148 (+ 267us) server_negotiation.cc:407] Connection header received
0808 08:20:24.400324 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.400328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.400399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.400486 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:24.400769 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.401498 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.402778 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.402988 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.403096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.403455 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.403547 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.403631 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.403723 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.403776 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:20:24.403959 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.399666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:38645 (local address 127.12.13.1:54513)
0808 08:20:24.400061 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.400076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:24.400228 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:24.400509 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:24.400512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:24.400520 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 08:20:24.400634 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.400640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.401611 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.401614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.402653 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:24.402665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.403112 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.403115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:24.403200 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.403260 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.403636 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.403646 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 08:20:24.403713 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 08:20:24.403797 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260808 08:20:24.427450 23738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:24.427634 23738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:24.429723 23738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Sending DeleteTablet for 3 replicas of tablet 7ae4ad382b0648f8908dd721efa0de24
I20260808 08:20:24.430513 24172 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae4ad382b0648f8908dd721efa0de24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50798
I20260808 08:20:24.430877 23967 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae4ad382b0648f8908dd721efa0de24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:39724
I20260808 08:20:24.431074 24384 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae4ad382b0648f8908dd721efa0de24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:57236
I20260808 08:20:24.431427 24612 tablet_replica.cc:330] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: stopping tablet replica
I20260808 08:20:24.431461 24610 tablet_replica.cc:330] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: stopping tablet replica
I20260808 08:20:24.431588 24610 raft_consensus.cc:2229] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:24.431615 24612 raft_consensus.cc:2229] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:24.431717 24610 raft_consensus.cc:2258] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.431968 24612 raft_consensus.cc:2258] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.432500 24610 ts_tablet_manager.cc:1905] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.432686 24612 ts_tablet_manager.cc:1905] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.435657 24612 ts_tablet_manager.cc:1918] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:20:24.435827 24612 log.cc:1196] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/7ae4ad382b0648f8908dd721efa0de24
I20260808 08:20:24.435846 24610 ts_tablet_manager.cc:1918] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:20:24.435922 24610 log.cc:1196] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/7ae4ad382b0648f8908dd721efa0de24
I20260808 08:20:24.436239 24610 ts_tablet_manager.cc:1939] T 7ae4ad382b0648f8908dd721efa0de24 P b60c036eb9564c9cb48834317220c923: Deleting consensus metadata
I20260808 08:20:24.436275 24612 ts_tablet_manager.cc:1939] T 7ae4ad382b0648f8908dd721efa0de24 P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting consensus metadata
I20260808 08:20:24.437014 23722 catalog_manager.cc:4841] TS c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645): tablet 7ae4ad382b0648f8908dd721efa0de24 (table test-table [id=2c8f49457e6c44cfae57bfac208bb547]) successfully deleted
I20260808 08:20:24.437014 23718 catalog_manager.cc:4841] TS b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393): tablet 7ae4ad382b0648f8908dd721efa0de24 (table test-table [id=2c8f49457e6c44cfae57bfac208bb547]) successfully deleted
I20260808 08:20:24.438071 24611 tablet_replica.cc:330] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: stopping tablet replica
I20260808 08:20:24.438254 24611 raft_consensus.cc:2229] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:24.438416 24611 raft_consensus.cc:2258] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.439055 24611 ts_tablet_manager.cc:1905] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.441709 24611 ts_tablet_manager.cc:1918] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:20:24.441797 24611 log.cc:1196] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/7ae4ad382b0648f8908dd721efa0de24
I20260808 08:20:24.442124 24611 ts_tablet_manager.cc:1939] T 7ae4ad382b0648f8908dd721efa0de24 P 19611cbe97ad42738e985b084fa8a1c8: Deleting consensus metadata
I20260808 08:20:24.443020 23722 catalog_manager.cc:4841] TS 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509): tablet 7ae4ad382b0648f8908dd721efa0de24 (table test-table [id=2c8f49457e6c44cfae57bfac208bb547]) successfully deleted
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[23679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 08:20:24.465699 23819 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:24.457978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.12.13.62:33841)
0808 08:20:24.458145 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.458151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.458376 (+ 225us) server_negotiation.cc:407] Connection header received
0808 08:20:24.458541 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.458544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.458594 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.458666 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:24.459656 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.460174 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.461401 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.461607 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.461682 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.465517 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58772: BlockingRecv error: recv got EOF from 127.0.0.1:58772 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[23679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM
I20260808 08:20:24.477077 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.467391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.12.13.62:33841)
0808 08:20:24.467561 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.467565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.467605 (+ 40us) server_negotiation.cc:407] Connection header received
0808 08:20:24.467744 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.467746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.467791 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.467863 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:24.468584 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.469102 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.470358 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.470573 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.470653 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.473408 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:24.473412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:24.473414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:24.473442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:24.475086 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:24.475686 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:24.475693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:24.475696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:24.475793 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:24.476095 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:24.476099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:24.476101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:24.476325 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:24.476483 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.476747 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.476877 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 08:20:24.480255 23738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:24.480836 23738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:24.486284 23967 tablet_service.cc:1467] Processing CreateTablet for tablet a1bff6d8f3cc47cebb154f18eac6f3cc (DEFAULT_TABLE table=test-table [id=60b6130901934139bbdff2d51d6117da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:24.486663 23967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1bff6d8f3cc47cebb154f18eac6f3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.488801 24525 tablet_bootstrap.cc:492] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Bootstrap starting.
I20260808 08:20:24.489063 24172 tablet_service.cc:1467] Processing CreateTablet for tablet a1bff6d8f3cc47cebb154f18eac6f3cc (DEFAULT_TABLE table=test-table [id=60b6130901934139bbdff2d51d6117da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:24.489383 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1bff6d8f3cc47cebb154f18eac6f3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.489903 24525 tablet_bootstrap.cc:654] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.491390 24525 tablet_bootstrap.cc:492] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: No bootstrap required, opened a new log
I20260808 08:20:24.491477 24525 ts_tablet_manager.cc:1397] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:24.491861 24525 raft_consensus.cc:348] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.491966 24525 raft_consensus.cc:374] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.491992 24525 raft_consensus.cc:729] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60c036eb9564c9cb48834317220c923, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.492110 24525 consensus_queue.cc:260] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.492197 24384 tablet_service.cc:1467] Processing CreateTablet for tablet a1bff6d8f3cc47cebb154f18eac6f3cc (DEFAULT_TABLE table=test-table [id=60b6130901934139bbdff2d51d6117da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:24.492341 24525 ts_tablet_manager.cc:1428] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:24.492472 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1bff6d8f3cc47cebb154f18eac6f3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.494776 24524 tablet_bootstrap.cc:492] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Bootstrap starting.
I20260808 08:20:24.494884 24523 tablet_bootstrap.cc:492] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Bootstrap starting.
I20260808 08:20:24.495689 24523 tablet_bootstrap.cc:654] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.495740 24524 tablet_bootstrap.cc:654] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.497062 24523 tablet_bootstrap.cc:492] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: No bootstrap required, opened a new log
I20260808 08:20:24.497139 24523 ts_tablet_manager.cc:1397] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:24.497557 24523 raft_consensus.cc:348] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.497663 24523 raft_consensus.cc:374] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.497701 24523 raft_consensus.cc:729] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.497830 24523 consensus_queue.cc:260] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.498118 24523 ts_tablet_manager.cc:1428] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:24.500600 24524 tablet_bootstrap.cc:492] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: No bootstrap required, opened a new log
I20260808 08:20:24.500705 24524 ts_tablet_manager.cc:1397] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 08:20:24.501605 24524 raft_consensus.cc:348] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.501744 24524 raft_consensus.cc:374] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.501785 24524 raft_consensus.cc:729] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19611cbe97ad42738e985b084fa8a1c8, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.501986 24524 consensus_queue.cc:260] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.502781 24524 ts_tablet_manager.cc:1428] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:24.596794 24532 raft_consensus.cc:482] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:24.596956 24532 raft_consensus.cc:504] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.597436 24532 leader_election.cc:288] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.598040 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bff6d8f3cc47cebb154f18eac6f3cc" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923" is_pre_election: true
I20260808 08:20:24.598040 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bff6d8f3cc47cebb154f18eac6f3cc" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8" is_pre_election: true
I20260808 08:20:24.598217 24419 raft_consensus.cc:2454] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:24.598215 23995 raft_consensus.cc:2454] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:24.598583 24101 leader_election.cc:302] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, b60c036eb9564c9cb48834317220c923, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.598816 24532 raft_consensus.cc:2781] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:24.598908 24532 raft_consensus.cc:482] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:24.598961 24532 raft_consensus.cc:3037] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.600021 24532 raft_consensus.cc:504] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.600471 24532 leader_election.cc:288] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Requested vote from peers b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.600844 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bff6d8f3cc47cebb154f18eac6f3cc" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8"
I20260808 08:20:24.600847 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bff6d8f3cc47cebb154f18eac6f3cc" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923"
I20260808 08:20:24.600970 24419 raft_consensus.cc:3037] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.601002 23995 raft_consensus.cc:3037] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.601982 24419 raft_consensus.cc:2454] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:24.602072 23995 raft_consensus.cc:2454] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:24.602288 24104 leader_election.cc:302] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.602481 24532 raft_consensus.cc:2781] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:24.602602 24532 raft_consensus.cc:686] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Becoming Leader. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Running, Role: LEADER
I20260808 08:20:24.602759 24532 consensus_queue.cc:237] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.604502 23738 catalog_manager.cc:5495] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:24.627045 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.623537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.12.13.2:38645)
0808 08:20:24.623718 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:24.623723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:24.623739 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:20:24.623818 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:24.623821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:24.623878 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:24.623941 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:24.624217 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.624761 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.625729 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:24.626072 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:24.626242 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:24.626500 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:24.626588 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:24.626661 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:24.626857 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 08:20:24.626916 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 08:20:24.629482 24419 raft_consensus.cc:1261] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.629482 23995 raft_consensus.cc:1261] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.630033 24532 consensus_queue.cc:1035] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.630205 24608 consensus_queue.cc:1035] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.637722 23736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:24.637923 23736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:24.640043 23736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Sending DeleteTablet for 3 replicas of tablet a1bff6d8f3cc47cebb154f18eac6f3cc
I20260808 08:20:24.640635 23967 tablet_service.cc:1514] Processing DeleteTablet for tablet a1bff6d8f3cc47cebb154f18eac6f3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:39724
I20260808 08:20:24.640792 24172 tablet_service.cc:1514] Processing DeleteTablet for tablet a1bff6d8f3cc47cebb154f18eac6f3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50798
I20260808 08:20:24.640892 24610 tablet_replica.cc:330] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: stopping tablet replica
I20260808 08:20:24.640982 24612 tablet_replica.cc:330] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: stopping tablet replica
I20260808 08:20:24.641049 24610 raft_consensus.cc:2229] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:24.641048 24384 tablet_service.cc:1514] Processing DeleteTablet for tablet a1bff6d8f3cc47cebb154f18eac6f3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:57236
I20260808 08:20:24.641085 24612 raft_consensus.cc:2229] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:24.641186 24610 raft_consensus.cc:2258] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.641258 24612 raft_consensus.cc:2258] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.641443 24611 tablet_replica.cc:330] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: stopping tablet replica
I20260808 08:20:24.641569 24611 raft_consensus.cc:2229] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:24.641723 24611 raft_consensus.cc:2258] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.642202 24610 ts_tablet_manager.cc:1905] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.642215 23736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 08:20:24.642478 24612 ts_tablet_manager.cc:1905] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 08:20:24.642581 23736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:24.643646 24611 ts_tablet_manager.cc:1905] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.644910 24612 ts_tablet_manager.cc:1918] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:24.644910 24610 ts_tablet_manager.cc:1918] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:24.644986 24612 log.cc:1196] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/a1bff6d8f3cc47cebb154f18eac6f3cc
I20260808 08:20:24.644985 24610 log.cc:1196] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/a1bff6d8f3cc47cebb154f18eac6f3cc
I20260808 08:20:24.645258 24612 ts_tablet_manager.cc:1939] T a1bff6d8f3cc47cebb154f18eac6f3cc P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting consensus metadata
I20260808 08:20:24.645272 24610 ts_tablet_manager.cc:1939] T a1bff6d8f3cc47cebb154f18eac6f3cc P b60c036eb9564c9cb48834317220c923: Deleting consensus metadata
I20260808 08:20:24.646217 24611 ts_tablet_manager.cc:1918] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:24.646206 23722 catalog_manager.cc:4841] TS c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645): tablet a1bff6d8f3cc47cebb154f18eac6f3cc (table test-table [id=60b6130901934139bbdff2d51d6117da]) successfully deleted
I20260808 08:20:24.646303 24611 log.cc:1196] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/a1bff6d8f3cc47cebb154f18eac6f3cc
I20260808 08:20:24.646811 24611 ts_tablet_manager.cc:1939] T a1bff6d8f3cc47cebb154f18eac6f3cc P 19611cbe97ad42738e985b084fa8a1c8: Deleting consensus metadata
I20260808 08:20:24.647089 23718 catalog_manager.cc:4841] TS b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393): tablet a1bff6d8f3cc47cebb154f18eac6f3cc (table test-table [id=60b6130901934139bbdff2d51d6117da]) successfully deleted
I20260808 08:20:24.647702 23722 catalog_manager.cc:4841] TS 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509): tablet a1bff6d8f3cc47cebb154f18eac6f3cc (table test-table [id=60b6130901934139bbdff2d51d6117da]) successfully deleted
I20260808 08:20:24.647727 23967 tablet_service.cc:1467] Processing CreateTablet for tablet f7f414a4d7e84dd0804dbb3ee257aeeb (DEFAULT_TABLE table=test-table [id=6e3daefe2804473880f2520d7f40ec89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:24.648002 23967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f414a4d7e84dd0804dbb3ee257aeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.648022 24384 tablet_service.cc:1467] Processing CreateTablet for tablet f7f414a4d7e84dd0804dbb3ee257aeeb (DEFAULT_TABLE table=test-table [id=6e3daefe2804473880f2520d7f40ec89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:24.648166 24172 tablet_service.cc:1467] Processing CreateTablet for tablet f7f414a4d7e84dd0804dbb3ee257aeeb (DEFAULT_TABLE table=test-table [id=6e3daefe2804473880f2520d7f40ec89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:24.648279 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f414a4d7e84dd0804dbb3ee257aeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.648427 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f414a4d7e84dd0804dbb3ee257aeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:24.650444 24524 tablet_bootstrap.cc:492] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Bootstrap starting.
I20260808 08:20:24.650520 24525 tablet_bootstrap.cc:492] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: Bootstrap starting.
I20260808 08:20:24.650745 24523 tablet_bootstrap.cc:492] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Bootstrap starting.
I20260808 08:20:24.651391 24524 tablet_bootstrap.cc:654] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.651407 24525 tablet_bootstrap.cc:654] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.651494 24523 tablet_bootstrap.cc:654] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:24.652611 24524 tablet_bootstrap.cc:492] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: No bootstrap required, opened a new log
I20260808 08:20:24.652673 24525 tablet_bootstrap.cc:492] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: No bootstrap required, opened a new log
I20260808 08:20:24.652707 24524 ts_tablet_manager.cc:1397] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:24.652748 24525 ts_tablet_manager.cc:1397] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:24.653014 24523 tablet_bootstrap.cc:492] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: No bootstrap required, opened a new log
I20260808 08:20:24.653126 24523 ts_tablet_manager.cc:1397] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:24.653159 24524 raft_consensus.cc:348] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.653159 24525 raft_consensus.cc:348] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.653283 24525 raft_consensus.cc:374] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.653283 24524 raft_consensus.cc:374] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.653321 24525 raft_consensus.cc:729] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60c036eb9564c9cb48834317220c923, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.653339 24524 raft_consensus.cc:729] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19611cbe97ad42738e985b084fa8a1c8, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.653456 24525 consensus_queue.cc:260] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.653486 24524 consensus_queue.cc:260] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.653596 24523 raft_consensus.cc:348] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.653733 24523 raft_consensus.cc:374] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:24.653741 24525 ts_tablet_manager.cc:1428] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 08:20:24.653774 24523 raft_consensus.cc:729] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Initialized, Role: FOLLOWER
I20260808 08:20:24.653952 24523 consensus_queue.cc:260] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.654213 24523 ts_tablet_manager.cc:1428] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 08:20:24.655934 24524 ts_tablet_manager.cc:1428] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:24.721170 24608 raft_consensus.cc:482] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:24.721360 24608 raft_consensus.cc:504] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.721900 24608 leader_election.cc:288] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.722337 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f414a4d7e84dd0804dbb3ee257aeeb" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923" is_pre_election: true
I20260808 08:20:24.722357 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f414a4d7e84dd0804dbb3ee257aeeb" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8" is_pre_election: true
I20260808 08:20:24.722515 24419 raft_consensus.cc:2454] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:24.722515 23995 raft_consensus.cc:2454] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 0.
I20260808 08:20:24.722837 24104 leader_election.cc:302] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19611cbe97ad42738e985b084fa8a1c8, b60c036eb9564c9cb48834317220c923, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.723018 24608 raft_consensus.cc:2781] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:24.723084 24608 raft_consensus.cc:482] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:24.723127 24608 raft_consensus.cc:3037] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.724228 24608 raft_consensus.cc:504] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.724560 24608 leader_election.cc:288] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Requested vote from peers b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393), 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509)
I20260808 08:20:24.724915 23995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f414a4d7e84dd0804dbb3ee257aeeb" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60c036eb9564c9cb48834317220c923"
I20260808 08:20:24.724942 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f414a4d7e84dd0804dbb3ee257aeeb" candidate_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19611cbe97ad42738e985b084fa8a1c8"
I20260808 08:20:24.725026 23995 raft_consensus.cc:3037] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.725090 24419 raft_consensus.cc:3037] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:24.726025 23995 raft_consensus.cc:2454] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:24.726163 24419 raft_consensus.cc:2454] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86ad0bbaf7f432baf89a8723dba8ec2 in term 1.
I20260808 08:20:24.726359 24101 leader_election.cc:302] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b60c036eb9564c9cb48834317220c923, c86ad0bbaf7f432baf89a8723dba8ec2; no voters:
I20260808 08:20:24.726572 24608 raft_consensus.cc:2781] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:24.726714 24608 raft_consensus.cc:686] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Becoming Leader. State: Replica: c86ad0bbaf7f432baf89a8723dba8ec2, State: Running, Role: LEADER
I20260808 08:20:24.726899 24608 consensus_queue.cc:237] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } }
I20260808 08:20:24.728648 23736 catalog_manager.cc:5495] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86ad0bbaf7f432baf89a8723dba8ec2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 38645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:24.751766 23995 raft_consensus.cc:1261] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.751801 24419 raft_consensus.cc:1261] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Refusing update from remote peer c86ad0bbaf7f432baf89a8723dba8ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:24.752246 24532 consensus_queue.cc:1035] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19611cbe97ad42738e985b084fa8a1c8" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.752430 24538 consensus_queue.cc:1035] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60c036eb9564c9cb48834317220c923" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:24.792338 23736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:24.792531 23736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:24.795042 23736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff7ea0f31cdd45e5a54b6b7b17ba3ae9: Sending DeleteTablet for 3 replicas of tablet f7f414a4d7e84dd0804dbb3ee257aeeb
I20260808 08:20:24.795756 24384 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f414a4d7e84dd0804dbb3ee257aeeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:57236
I20260808 08:20:24.795825 24172 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f414a4d7e84dd0804dbb3ee257aeeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50798
I20260808 08:20:24.796041 24612 tablet_replica.cc:330] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: stopping tablet replica
I20260808 08:20:24.796070 23967 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f414a4d7e84dd0804dbb3ee257aeeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:24 UTC) from {username='oryx', principal='oryx/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:39724
I20260808 08:20:24.796183 24612 raft_consensus.cc:2229] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:24.796276 24610 tablet_replica.cc:330] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: stopping tablet replica
I20260808 08:20:24.796402 24612 raft_consensus.cc:2258] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.796406 24610 raft_consensus.cc:2229] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:24.796526 24610 raft_consensus.cc:2258] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.796895 24611 tablet_replica.cc:330] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: stopping tablet replica
I20260808 08:20:24.797083 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 23850
I20260808 08:20:24.797422 24610 ts_tablet_manager.cc:1905] T f7f414a4d7e84dd0804dbb3ee257aeeb P b60c036eb9564c9cb48834317220c923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.803439 24612 ts_tablet_manager.cc:1905] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:24.810899 24612 ts_tablet_manager.cc:1918] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:20:24.811031 24612 log.cc:1196] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/f7f414a4d7e84dd0804dbb3ee257aeeb
I20260808 08:20:24.811384 24612 ts_tablet_manager.cc:1939] T f7f414a4d7e84dd0804dbb3ee257aeeb P c86ad0bbaf7f432baf89a8723dba8ec2: Deleting consensus metadata
I20260808 08:20:24.812402 23722 catalog_manager.cc:4841] TS c86ad0bbaf7f432baf89a8723dba8ec2 (127.12.13.2:38645): tablet f7f414a4d7e84dd0804dbb3ee257aeeb (table test-table [id=6e3daefe2804473880f2520d7f40ec89]) successfully deleted
I20260808 08:20:24.814075 24611 raft_consensus.cc:2229] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:24.814337 24611 raft_consensus.cc:2258] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:24.815187 24611 ts_tablet_manager.cc:1905] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 08:20:24.817090 23718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:35393 (error 108)
I20260808 08:20:24.818008 24611 ts_tablet_manager.cc:1918] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 08:20:24.818090 24611 log.cc:1196] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/f7f414a4d7e84dd0804dbb3ee257aeeb
I20260808 08:20:24.818360 24611 ts_tablet_manager.cc:1939] T f7f414a4d7e84dd0804dbb3ee257aeeb P 19611cbe97ad42738e985b084fa8a1c8: Deleting consensus metadata
I20260808 08:20:24.818416 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:24.817910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:35393 (local address 127.0.0.1:39728)
0808 08:20:24.818167 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:24.818261 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
W20260808 08:20:24.818567 23718 catalog_manager.cc:4584] TS b60c036eb9564c9cb48834317220c923 (127.12.13.1:35393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7f414a4d7e84dd0804dbb3ee257aeeb: Network error: Client connection negotiation failed: client connection to 127.12.13.1:35393: connect: Connection refused (error 111)
I20260808 08:20:24.819165 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 24076
I20260808 08:20:24.819162 23722 catalog_manager.cc:4841] TS 19611cbe97ad42738e985b084fa8a1c8 (127.12.13.3:45509): tablet f7f414a4d7e84dd0804dbb3ee257aeeb (table test-table [id=6e3daefe2804473880f2520d7f40ec89]) successfully deleted
I20260808 08:20:24.837464 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 24286
I20260808 08:20:24.853266 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 23697
2026-08-08T08:20:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3511 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[24663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[24663](info): set up 2 sockets
Aug 08 08:20:24 dist-test-slave-6ntk krb5kdc[24663](info): commencing operation
krb5kdc: starting...
W20260808 08:20:26.113859 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:26 dist-test-slave-6ntk krb5kdc[24663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:26.253191 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:45737
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:26.373772 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:26.374181 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:26.374262 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:26.378667 24689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:26.378760 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:26.378798 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:26.378830 24689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:26.378870 24689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:26.384430 24689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:26.385576 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:26.386744 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:26.395762 24699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:26.395805 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:26.395948 24689 server_base.cc:1034] running on GCE node
W20260808 08:20:26.395751 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:26.396656 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:26.397820 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:26.399044 24689 hybrid_clock.cc:648] HybridClock initialized: now 1786177226399024 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T08:20:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1562 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:26 dist-test-slave-6ntk krb5kdc[24712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:26 dist-test-slave-6ntk krb5kdc[24712](info): set up 2 sockets
Aug 08 08:20:26 dist-test-slave-6ntk krb5kdc[24712](info): commencing operation
krb5kdc: starting...
W20260808 08:20:27.716737 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:27 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:27.853732 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:44269
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:44269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:27.971812 24733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:27.972148 24733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:27.972218 24733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:27.976294 24733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:27.976413 24733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:27.976449 24733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:27.976491 24733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:27.976528 24733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:27.982120 24733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:44269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:44269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:27.983274 24733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:27.984366 24733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:27.993257 24745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:27.993278 24741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:27.993257 24742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:27.993649 24733 server_base.cc:1034] running on GCE node
I20260808 08:20:27.994295 24733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:27.995414 24733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:27.996608 24733 hybrid_clock.cc:648] HybridClock initialized: now 1786177227996588 us; error 32 us; skew 500 ppm
Aug 08 08:20:27 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177227, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:27.999445 24733 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:28.000432 24733 webserver.cc:460] Webserver started at http://127.12.13.62:45517/ using document root <none> and password file <none>
I20260808 08:20:28.001024 24733 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:28.001118 24733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:28.001366 24733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:28.003541 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "a816ef17ab6e460e9ee4f76869045c94"
format_stamp: "Formatted at 2026-08-08 08:20:27 on dist-test-slave-6ntk"
server_key: "f40f67e51336c64ece894d001b25de72"
server_key_iv: "d85245c55fdb82c94bdfaa1133850093"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.004112 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "a816ef17ab6e460e9ee4f76869045c94"
format_stamp: "Formatted at 2026-08-08 08:20:27 on dist-test-slave-6ntk"
server_key: "f40f67e51336c64ece894d001b25de72"
server_key_iv: "d85245c55fdb82c94bdfaa1133850093"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.008056 24733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:28.010572 24755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.011868 24733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:28.012023 24733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "a816ef17ab6e460e9ee4f76869045c94"
format_stamp: "Formatted at 2026-08-08 08:20:27 on dist-test-slave-6ntk"
server_key: "f40f67e51336c64ece894d001b25de72"
server_key_iv: "d85245c55fdb82c94bdfaa1133850093"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.012140 24733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:28.029235 24733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:28.032375 24733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:28.032601 24733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:28.040625 24733 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:44269
I20260808 08:20:28.040650 24840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:44269 every 8 connection(s)
I20260808 08:20:28.041692 24733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:28.044798 24841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:28.050823 24841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Bootstrap starting.
I20260808 08:20:28.051818 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 24733
I20260808 08:20:28.051990 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:28.052284 12340 external_mini_cluster.cc:1428] Setting key de254dcf391cec64e4a3672a310ff458
I20260808 08:20:28.053990 24841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:28.054773 24841 log.cc:824] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:28.057113 24841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: No bootstrap required, opened a new log
I20260808 08:20:28.061442 24841 raft_consensus.cc:348] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:28.061712 24841 raft_consensus.cc:374] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:28.061810 24841 raft_consensus.cc:729] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a816ef17ab6e460e9ee4f76869045c94, State: Initialized, Role: FOLLOWER
I20260808 08:20:28.062357 24841 consensus_queue.cc:260] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:28.062522 24841 raft_consensus.cc:388] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:28.062605 24841 raft_consensus.cc:482] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:28.062722 24841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:28.064142 24841 raft_consensus.cc:504] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:28.064549 24841 leader_election.cc:302] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a816ef17ab6e460e9ee4f76869045c94; no voters:
I20260808 08:20:28.064934 24841 leader_election.cc:288] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:28.065056 24849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:28.065292 24849 raft_consensus.cc:686] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 LEADER]: Becoming Leader. State: Replica: a816ef17ab6e460e9ee4f76869045c94, State: Running, Role: LEADER
I20260808 08:20:28.065722 24849 consensus_queue.cc:237] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:28.066350 24841 sys_catalog.cc:564] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:28.067214 24853 sys_catalog.cc:455] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a816ef17ab6e460e9ee4f76869045c94. Latest consensus state: current_term: 1 leader_uuid: "a816ef17ab6e460e9ee4f76869045c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } } }
I20260808 08:20:28.067628 24853 sys_catalog.cc:458] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:28.068452 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.053519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.12.13.62:44269)
0808 08:20:28.054022 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:28.054040 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:28.054075 (+ 35us) server_negotiation.cc:407] Connection header received
0808 08:20:28.054781 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:28.054799 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:28.055196 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:28.055487 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:28.056303 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.057776 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.058696 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.059441 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.059608 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.062408 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:28.062415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:28.062432 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:28.062469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:28.064571 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.065109 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.065118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.065128 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.065301 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.065680 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.065683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.065684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.066278 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:28.066505 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:28.066925 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 08:20:28.067270 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":156,"threads_started":1}
I20260808 08:20:28.067647 24850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a816ef17ab6e460e9ee4f76869045c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } } }
I20260808 08:20:28.068867 24850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: This master's current role is: LEADER
W20260808 08:20:28.072325 24875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:20:28.072394 24875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:20:28.072470 24862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:28.073218 24862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:28.079617 24862 catalog_manager.cc:1347] Generated new cluster ID: 7510b78a08554759ad2c69e0240557ea
I20260808 08:20:28.079780 24862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:28.089346 24862 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:28.090355 24862 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:28.096097 24862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Generated new TSK 0
I20260808 08:20:28.096787 24862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:20:28.134548 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44269
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:28.259394 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:28.259683 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:28.259727 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:28.263885 24882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:28.263948 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:28.264006 24882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:28.269608 24882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:28.270735 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:28.271827 24882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:28.282348 24890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:28.282366 24891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:28.282348 24894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:28.282822 24882 server_base.cc:1034] running on GCE node
I20260808 08:20:28.283305 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:28.284047 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:28.285236 24882 hybrid_clock.cc:648] HybridClock initialized: now 1786177228285214 us; error 38 us; skew 500 ppm
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:28.288173 24882 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:20:28.289143 24882 webserver.cc:460] Webserver started at http://127.12.13.1:42571/ using document root <none> and password file <none>
I20260808 08:20:28.289763 24882 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:28.289827 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:28.290088 24882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:28.292253 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "2516d7ec6fb2407893a61204a5e64e96"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "35a97e769f58240cf1c81d2006a9c65b"
server_key_iv: "09a21ab8eeb55332ad2955f5ee39ce91"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.292825 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "2516d7ec6fb2407893a61204a5e64e96"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "35a97e769f58240cf1c81d2006a9c65b"
server_key_iv: "09a21ab8eeb55332ad2955f5ee39ce91"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.296666 24882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:20:28.299248 24906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.300518 24882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:28.300653 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "2516d7ec6fb2407893a61204a5e64e96"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "35a97e769f58240cf1c81d2006a9c65b"
server_key_iv: "09a21ab8eeb55332ad2955f5ee39ce91"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.300765 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:28.335630 24882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:28.339002 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:28.339239 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:28.339906 24882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:28.340865 24882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:28.340942 24882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.341019 24882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:28.341066 24882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.351352 24882 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:44543
I20260808 08:20:28.351383 25081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:44543 every 8 connection(s)
I20260808 08:20:28.352370 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:28.353124 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 24882
I20260808 08:20:28.353284 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:28.353663 12340 external_mini_cluster.cc:1428] Setting key 1f83545cb5720e26dbe2370a2c83ec71
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:28.368913 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.354647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:45455 (local address 127.12.13.62:44269)
0808 08:20:28.354791 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:28.354794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:28.356325 (+ 1531us) server_negotiation.cc:407] Connection header received
0808 08:20:28.357724 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:28.357729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:28.357812 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:28.357951 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:28.359202 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.360076 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.361339 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.361774 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.361951 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.364820 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:28.364827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:28.364831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:28.364868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:28.366540 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.367238 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.367241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.367243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.367322 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.367645 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.367647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.367648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.367920 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:28.368001 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:28.368617 (+ 616us) server_negotiation.cc:299] Negotiation successful
0808 08:20:28.368724 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:20:28.370132 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.355182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:45455)
0808 08:20:28.356136 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:28.356201 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:28.357470 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:28.358128 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:28.358136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:28.358510 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:28.359037 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:28.359051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.360213 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.360216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:28.361208 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:28.361219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.361994 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.361997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:28.362147 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.362866 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:28.362883 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:28.364620 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:28.366658 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:28.366664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:28.366677 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:28.367132 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:28.367409 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:28.367411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:28.367413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:28.367560 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:28.368017 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:28.368023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:28.368382 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 08:20:28.368564 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 08:20:28.368860 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":161,"threads_started":1}
I20260808 08:20:28.371671 25082 heartbeater.cc:347] Connected to a master server at 127.12.13.62:44269
I20260808 08:20:28.371999 25082 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:28.372666 25082 heartbeater.cc:511] Master 127.12.13.62:44269 requested a full tablet report, sending...
I20260808 08:20:28.374842 24778 ts_manager.cc:194] Registered new tserver with Master: 2516d7ec6fb2407893a61204a5e64e96 (127.12.13.1:44543)
I20260808 08:20:28.376438 24778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:45455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:20:28.390185 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44269
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:28.513247 25090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:28.513545 25090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:28.513589 25090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:28.519470 25090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:28.519544 25090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:28.519637 25090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:28.525231 25090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:28.526520 25090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:28.527730 25090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:28.537997 25100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:28.537997 25101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:28.538096 25103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:28.538184 25090 server_base.cc:1034] running on GCE node
I20260808 08:20:28.538805 25090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:28.539526 25090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:28.540719 25090 hybrid_clock.cc:648] HybridClock initialized: now 1786177228540702 us; error 29 us; skew 500 ppm
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:28.543699 25090 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:20:28.544652 25090 webserver.cc:460] Webserver started at http://127.12.13.2:32995/ using document root <none> and password file <none>
I20260808 08:20:28.545269 25090 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:28.545401 25090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:28.545655 25090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:28.547923 25090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "c30c2e9c740f480ba84a562a94815865"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "136e5bb805c965d02b70c0cc7f81a7bf"
server_key_iv: "134f4343762d74a69b08f1daf2c87aa3"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.548544 25090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "c30c2e9c740f480ba84a562a94815865"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "136e5bb805c965d02b70c0cc7f81a7bf"
server_key_iv: "134f4343762d74a69b08f1daf2c87aa3"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.552596 25090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:20:28.555316 25116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.556667 25090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:20:28.556823 25090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "c30c2e9c740f480ba84a562a94815865"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "136e5bb805c965d02b70c0cc7f81a7bf"
server_key_iv: "134f4343762d74a69b08f1daf2c87aa3"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.556945 25090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:28.575845 25090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:28.579646 25090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:28.579891 25090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:28.580600 25090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:28.581570 25090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:28.581646 25090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.581750 25090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:28.581794 25090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.592396 25090 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:42551
I20260808 08:20:28.592417 25303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:42551 every 8 connection(s)
I20260808 08:20:28.593396 25090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:28.598205 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 25090
I20260808 08:20:28.598351 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:28.598677 12340 external_mini_cluster.cc:1428] Setting key 394471922fe34ffa015aeae655ab8d95
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:28.609900 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.595472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:53325 (local address 127.12.13.62:44269)
0808 08:20:28.595631 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:28.595635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:28.596955 (+ 1320us) server_negotiation.cc:407] Connection header received
0808 08:20:28.597857 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:28.597860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:28.597991 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:28.598091 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:28.599766 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.600495 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.602281 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.602499 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.602620 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.605630 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:28.605636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:28.605643 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:28.605682 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:28.607556 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.608116 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.608122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.608126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.608236 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.608582 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.608588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.608591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.608865 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:28.608979 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:28.609503 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 08:20:28.609645 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:20:28.610855 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.595977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:53325)
0808 08:20:28.596774 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:28.596820 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:28.597615 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:28.598344 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:28.598354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:28.598890 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:28.599602 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:28.599620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.600642 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.600647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:28.602151 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:28.602167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.602667 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.602671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:28.602863 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.603572 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:28.603596 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:28.605435 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:28.607694 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:28.607700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:28.607713 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:28.608005 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:28.608337 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:28.608340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:28.608341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:28.608481 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:28.608981 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:28.608987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:28.609237 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 08:20:28.609411 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 08:20:28.609685 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":132,"threads_started":1}
I20260808 08:20:28.612185 25304 heartbeater.cc:347] Connected to a master server at 127.12.13.62:44269
I20260808 08:20:28.612490 25304 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:28.613030 25304 heartbeater.cc:511] Master 127.12.13.62:44269 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:20:28.614311 24778 ts_manager.cc:194] Registered new tserver with Master: c30c2e9c740f480ba84a562a94815865 (127.12.13.2:42551)
I20260808 08:20:28.615147 24778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:53325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:20:28.633226 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44269
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:28.750198 25316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:28.750540 25316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:28.750620 25316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:28.754650 25316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:28.754741 25316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:28.754833 25316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:28.760286 25316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:28.761440 25316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:28.762552 25316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:28.772729 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:28.772758 25328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:28.772732 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:28.773164 25316 server_base.cc:1034] running on GCE node
I20260808 08:20:28.773610 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:28.774374 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:28.775573 25316 hybrid_clock.cc:648] HybridClock initialized: now 1786177228775557 us; error 29 us; skew 500 ppm
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:28.778620 25316 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:20:28.779536 25316 webserver.cc:460] Webserver started at http://127.12.13.3:34145/ using document root <none> and password file <none>
I20260808 08:20:28.780117 25316 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:28.780177 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:28.780391 25316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:28.782704 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "affb8b64a3084ad7b3fa1a190bcf9455"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "a9a8087cdbcbfeecb29e257439a80f84"
server_key_iv: "48d5f1b87c49d831bd3ca59831c71359"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.783319 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "affb8b64a3084ad7b3fa1a190bcf9455"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "a9a8087cdbcbfeecb29e257439a80f84"
server_key_iv: "48d5f1b87c49d831bd3ca59831c71359"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.787371 25316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:28.790086 25340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.791464 25316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:28.791623 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "affb8b64a3084ad7b3fa1a190bcf9455"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "a9a8087cdbcbfeecb29e257439a80f84"
server_key_iv: "48d5f1b87c49d831bd3ca59831c71359"
server_key_version: "encryptionkey@0"
I20260808 08:20:28.791747 25316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:28.814652 25316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:28.818099 25316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:28.818318 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:28.818967 25316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:28.819869 25316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:28.819947 25316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.820029 25316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:28.820073 25316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:28.829177 25316 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:44817
I20260808 08:20:28.829209 25533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:44817 every 8 connection(s)
I20260808 08:20:28.830150 25316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:28.831434 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 25316
I20260808 08:20:28.831579 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:28.831887 12340 external_mini_cluster.cc:1428] Setting key 83822256f1e1d4c698b40f5e138225ae
Aug 08 08:20:28 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:28.844226 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.832257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:48911 (local address 127.12.13.62:44269)
0808 08:20:28.832410 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:28.832414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:28.833766 (+ 1352us) server_negotiation.cc:407] Connection header received
0808 08:20:28.834805 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:28.834807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:28.834862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:28.834927 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:28.836060 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.836582 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.837857 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.838077 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.838153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.840816 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:28.840819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:28.840821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:28.840865 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:28.842413 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.842890 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.842893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.842894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.842972 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:28.843265 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:28.843267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:28.843269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:28.843445 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:28.843521 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:28.843990 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 08:20:28.844096 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:20:28.845516 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:28.832773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:48911)
0808 08:20:28.833603 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:28.833657 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:28.834615 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:28.835064 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:28.835070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:28.835439 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:28.835936 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:28.835950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.836690 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.836692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:28.837745 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:28.837755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:28.838207 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:28.838210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:28.838380 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:28.839037 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:28.839054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:28.840645 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:28.842518 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:28.842522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:28.842536 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:28.842791 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:28.843053 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:28.843055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:28.843057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:28.843186 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:28.843532 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:28.843537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:28.843769 (+ 232us) client_negotiation.cc:769] Sending connection context
0808 08:20:28.843937 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 08:20:28.844190 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":173,"threads_started":1}
I20260808 08:20:28.846763 25537 heartbeater.cc:347] Connected to a master server at 127.12.13.62:44269
I20260808 08:20:28.846980 25537 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:28.847458 25537 heartbeater.cc:511] Master 127.12.13.62:44269 requested a full tablet report, sending...
I20260808 08:20:28.848536 24778 ts_manager.cc:194] Registered new tserver with Master: affb8b64a3084ad7b3fa1a190bcf9455 (127.12.13.3:44817)
I20260808 08:20:28.849135 24778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:48911
I20260808 08:20:28.858333 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab.
Entry for principal oryx/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab.
WARNING: no policy specified for oryx/127.12.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab.
Entry for principal oryx/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab.
WARNING: no policy specified for oryx/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab.
Entry for principal oryx/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab.
I20260808 08:20:28.945508 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 24882
I20260808 08:20:28.960729 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 25090
I20260808 08:20:28.973208 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 25316
I20260808 08:20:28.999744 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 24733
I20260808 08:20:29.015944 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:44269
--webserver_interface=127.12.13.62
--webserver_port=45517
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:44269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:29.135665 25553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:29.135962 25553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:29.136041 25553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:29.140218 25553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:29.140373 25553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:29.140427 25553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:29.140486 25553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:29.140543 25553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:29.146137 25553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:44269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:44269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=45517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:29.147302 25553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:29.148424 25553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:29.157332 25568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.157389 25563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.157332 25564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:29.157666 25553 server_base.cc:1034] running on GCE node
I20260808 08:20:29.158465 25553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:29.159698 25553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:29.160897 25553 hybrid_clock.cc:648] HybridClock initialized: now 1786177229160881 us; error 30 us; skew 500 ppm
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:29.163892 25553 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:29.164892 25553 webserver.cc:460] Webserver started at http://127.12.13.62:45517/ using document root <none> and password file <none>
I20260808 08:20:29.165448 25553 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:29.165524 25553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:29.169582 25553 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:29.171967 25576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.173408 25553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 08:20:29.173549 25553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "a816ef17ab6e460e9ee4f76869045c94"
format_stamp: "Formatted at 2026-08-08 08:20:27 on dist-test-slave-6ntk"
server_key: "f40f67e51336c64ece894d001b25de72"
server_key_iv: "d85245c55fdb82c94bdfaa1133850093"
server_key_version: "encryptionkey@0"
I20260808 08:20:29.174180 25553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:29.190657 25553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:29.194160 25553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:29.194391 25553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:29.202078 25553 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:44269
I20260808 08:20:29.202102 25650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:44269 every 8 connection(s)
I20260808 08:20:29.203279 25553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:29.207834 25651 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:20:29.209587 25651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Bootstrap starting.
I20260808 08:20:29.213495 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 25553
I20260808 08:20:29.214076 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:44543
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:44269
--webserver_port=42571
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:20:29.228166 25651 log.cc:824] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:29.235720 25651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:20:29.236133 25651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Bootstrap complete.
I20260808 08:20:29.240543 25651 raft_consensus.cc:348] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:29.240968 25651 raft_consensus.cc:729] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a816ef17ab6e460e9ee4f76869045c94, State: Initialized, Role: FOLLOWER
I20260808 08:20:29.241506 25651 consensus_queue.cc:260] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:29.241624 25651 raft_consensus.cc:388] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:29.241714 25651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:29.241815 25651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 1 FOLLOWER]: Advancing to term 2
I20260808 08:20:29.243856 25651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:29.244216 25651 leader_election.cc:302] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a816ef17ab6e460e9ee4f76869045c94; no voters:
I20260808 08:20:29.244587 25651 leader_election.cc:288] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 08:20:29.244676 25656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 08:20:29.244868 25656 raft_consensus.cc:686] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [term 2 LEADER]: Becoming Leader. State: Replica: a816ef17ab6e460e9ee4f76869045c94, State: Running, Role: LEADER
I20260808 08:20:29.245247 25656 consensus_queue.cc:237] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } }
I20260808 08:20:29.245968 25651 sys_catalog.cc:564] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:29.246980 25657 sys_catalog.cc:455] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a816ef17ab6e460e9ee4f76869045c94. Latest consensus state: current_term: 2 leader_uuid: "a816ef17ab6e460e9ee4f76869045c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } } }
I20260808 08:20:29.247030 25659 sys_catalog.cc:455] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a816ef17ab6e460e9ee4f76869045c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a816ef17ab6e460e9ee4f76869045c94" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44269 } } }
I20260808 08:20:29.247115 25657 sys_catalog.cc:458] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:29.247143 25659 sys_catalog.cc:458] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:29.247629 25668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:29.251214 25668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:29.251950 25668 catalog_manager.cc:1259] Loaded cluster ID: 7510b78a08554759ad2c69e0240557ea
I20260808 08:20:29.252017 25668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:29.254407 25668 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:29.255079 25668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a816ef17ab6e460e9ee4f76869045c94: Loaded TSK: 0
I20260808 08:20:29.256307 25668 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 08:20:29.340711 25653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:29.341049 25653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:29.341126 25653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:29.345796 25653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:29.345958 25653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:29.346091 25653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:29.352061 25653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.1.keytab
--principal=oryx/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:44543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=42571
--tserver_master_addrs=127.12.13.62:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:29.353276 25653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:29.354429 25653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:29.364874 25688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.364886 25685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.365245 25692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:29.365509 25653 server_base.cc:1034] running on GCE node
I20260808 08:20:29.366067 25653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:29.366916 25653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:29.368129 25653 hybrid_clock.cc:648] HybridClock initialized: now 1786177229368092 us; error 49 us; skew 500 ppm
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:29.371721 25653 init.cc:377] Logged in from keytab as oryx/127.12.13.1@KRBTEST.COM (short username oryx)
I20260808 08:20:29.372759 25653 webserver.cc:460] Webserver started at http://127.12.13.1:42571/ using document root <none> and password file <none>
I20260808 08:20:29.373453 25653 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:29.373551 25653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:29.378814 25653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:29.381505 25704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.383141 25653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:29.383287 25653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "2516d7ec6fb2407893a61204a5e64e96"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "35a97e769f58240cf1c81d2006a9c65b"
server_key_iv: "09a21ab8eeb55332ad2955f5ee39ce91"
server_key_version: "encryptionkey@0"
I20260808 08:20:29.384119 25653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:29.404124 25653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:29.407428 25653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:29.407595 25653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:29.408185 25653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:29.409019 25653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:29.409067 25653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.409109 25653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:29.409124 25653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.418808 25653 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:44543
I20260808 08:20:29.418845 25854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:44543 every 8 connection(s)
I20260808 08:20:29.419879 25653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:29.422680 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 25653
I20260808 08:20:29.423164 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:42551
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:44269
--webserver_port=32995
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 08:20:29.435701 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.422209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:58443 (local address 127.12.13.62:44269)
0808 08:20:29.422814 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.422835 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.423419 (+ 584us) server_negotiation.cc:407] Connection header received
0808 08:20:29.424486 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.424504 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.424923 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.425269 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.426992 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.428035 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.429979 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.430485 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.430612 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.434918 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:58443: BlockingRecv error: recv got EOF from 127.12.13.1:58443 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":141,"threads_started":1}
I20260808 08:20:29.435710 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.422342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:58443)
0808 08:20:29.423202 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.423260 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.424052 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.425434 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.425445 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.426037 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.426765 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.426790 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.428196 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.428203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.429806 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.429823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.430675 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.430682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.430934 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.431625 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.431653 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.434682 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":140,"threads_started":1}
W20260808 08:20:29.436281 25857 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:44269 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.13.62:44269: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:29.442304 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.436787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:56837)
0808 08:20:29.436908 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.436925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.437007 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.437293 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.437295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.437461 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.437548 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.437553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.438389 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.438394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.438947 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.438955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.439419 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.439423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.439523 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.440042 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.440053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.442095 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 08:20:29.442323 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.436878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56837 (local address 127.12.13.62:44269)
0808 08:20:29.437010 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.437014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.437029 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:20:29.437099 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.437102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.437152 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.437279 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.437684 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.438277 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.439071 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.439295 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.439403 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.442221 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:56837: BlockingRecv error: recv got EOF from 127.12.13.1:56837 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:29.448415 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.442822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:59901)
0808 08:20:29.442967 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.442982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.443060 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.443264 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.443266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.443430 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.443519 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.443525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.444176 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.444179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.445026 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.445037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.445507 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.445510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.445592 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.446242 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.446257 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.448286 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 08:20:29.448661 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.442947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59901 (local address 127.12.13.62:44269)
0808 08:20:29.443079 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.443082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.443095 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:20:29.443126 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.443128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.443163 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.443227 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.443626 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.444080 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.445154 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.445398 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.445477 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.448515 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:59901: BlockingRecv error: recv got EOF from 127.12.13.1:59901 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 08:20:29.448793 25857 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:20:29.549404 25864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:29.549674 25864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:29.549717 25864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:29.553892 25864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:29.553957 25864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:29.554052 25864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:29.559561 25864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.2.keytab
--principal=oryx/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:42551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=32995
--tserver_master_addrs=127.12.13.62:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:29.560912 25864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:29.562077 25864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:29.575780 25870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.575769 25869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.575740 25872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:29.580605 25864 server_base.cc:1034] running on GCE node
I20260808 08:20:29.581220 25864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:29.582228 25864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:29.583431 25864 hybrid_clock.cc:648] HybridClock initialized: now 1786177229583378 us; error 80 us; skew 500 ppm
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:29.586542 25864 init.cc:377] Logged in from keytab as oryx/127.12.13.2@KRBTEST.COM (short username oryx)
I20260808 08:20:29.587594 25864 webserver.cc:460] Webserver started at http://127.12.13.2:32995/ using document root <none> and password file <none>
I20260808 08:20:29.588253 25864 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:29.588372 25864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:29.592785 25864 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:29.595160 25882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.596421 25864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:29.596570 25864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "c30c2e9c740f480ba84a562a94815865"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "136e5bb805c965d02b70c0cc7f81a7bf"
server_key_iv: "134f4343762d74a69b08f1daf2c87aa3"
server_key_version: "encryptionkey@0"
I20260808 08:20:29.597168 25864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:29.611953 25864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:29.615149 25864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:29.615370 25864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:29.616039 25864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:29.616991 25864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:29.617069 25864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.617149 25864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:29.617201 25864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.627040 25864 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:42551
I20260808 08:20:29.627112 26052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:42551 every 8 connection(s)
I20260808 08:20:29.628206 25864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:29.632056 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 25864
I20260808 08:20:29.632565 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:44817
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:44269
--webserver_port=34145
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 08:20:29.639981 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.630270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:38123 (local address 127.12.13.62:44269)
0808 08:20:29.630430 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.630434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.631574 (+ 1140us) server_negotiation.cc:407] Connection header received
0808 08:20:29.632559 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.632563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.632629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.632734 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.634118 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.634648 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.635809 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.636035 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.636147 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.639795 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:38123: BlockingRecv error: recv got EOF from 127.12.13.2:38123 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:20:29.640465 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.630706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:38123)
0808 08:20:29.631401 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.631444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.632336 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.633012 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.633018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.633397 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.633960 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.633976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.634779 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.634782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.635670 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.635679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.636192 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.636195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.636373 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.637004 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.637022 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.639554 (+ 2532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":124,"threads_started":1}
W20260808 08:20:29.641105 26053 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:44269 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.13.62:44269: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:29.646965 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.641635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:43731)
0808 08:20:29.641758 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.641775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.641886 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.642135 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.642138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.642310 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.642403 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.642409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.643125 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.643130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.643663 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.643670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.644104 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.644108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.644193 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.644755 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.644767 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.646783 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 08:20:29.647018 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.641756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:43731 (local address 127.12.13.62:44269)
0808 08:20:29.641893 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.641897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.641913 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:20:29.641981 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.641984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.642033 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.642119 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.642517 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.643016 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.643775 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.643994 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.644103 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.646909 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:43731: BlockingRecv error: recv got EOF from 127.12.13.2:43731 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:29.654289 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.647818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:54777)
0808 08:20:29.647927 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.647941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.648038 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.648285 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.648287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.648420 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.648504 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.648509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.649521 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.649527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.650452 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.650464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.651462 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.651467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.651542 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.652094 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.652114 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.654133 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 08:20:29.654522 26053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:20:29.654681 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.647897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:54777 (local address 127.12.13.62:44269)
0808 08:20:29.648051 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.648055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.648098 (+ 43us) server_negotiation.cc:407] Connection header received
0808 08:20:29.648138 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.648141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.648187 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.648261 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.648617 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.649414 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.650709 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.651118 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.651290 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.654308 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:54777: BlockingRecv error: recv got EOF from 127.12.13.2:54777 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 08:20:29.761930 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:29.762234 26057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:29.762279 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:29.766419 26057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:29.766487 26057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:29.766547 26057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:29.772004 26057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/krb5kdc/oryx_127.12.13.3.keytab
--principal=oryx/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:44817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=34145
--tserver_master_addrs=127.12.13.62:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:29.773134 26057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:29.774276 26057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:29.784955 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:29.785037 26057 server_base.cc:1034] running on GCE node
W20260808 08:20:29.784933 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:29.785219 26065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:29.785684 26057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:29.786422 26057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:29.787595 26057 hybrid_clock.cc:648] HybridClock initialized: now 1786177229787581 us; error 26 us; skew 500 ppm
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:29.790800 26057 init.cc:377] Logged in from keytab as oryx/127.12.13.3@KRBTEST.COM (short username oryx)
I20260808 08:20:29.791821 26057 webserver.cc:460] Webserver started at http://127.12.13.3:34145/ using document root <none> and password file <none>
I20260808 08:20:29.792443 26057 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:29.792558 26057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:29.796898 26057 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:29.799373 26077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.800738 26057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:20:29.800894 26057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "affb8b64a3084ad7b3fa1a190bcf9455"
format_stamp: "Formatted at 2026-08-08 08:20:28 on dist-test-slave-6ntk"
server_key: "a9a8087cdbcbfeecb29e257439a80f84"
server_key_iv: "48d5f1b87c49d831bd3ca59831c71359"
server_key_version: "encryptionkey@0"
I20260808 08:20:29.801504 26057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:29.814685 26057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:29.818426 26057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:29.818593 26057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:29.819171 26057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:29.820009 26057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:29.820056 26057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.820098 26057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:29.820113 26057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:29.829131 26057 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:44817
I20260808 08:20:29.829162 26242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:44817 every 8 connection(s)
I20260808 08:20:29.830188 26057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:29.832360 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 26057
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 08:20:29.841410 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.832079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:59175 (local address 127.12.13.62:44269)
0808 08:20:29.832222 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.832226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.833554 (+ 1328us) server_negotiation.cc:407] Connection header received
0808 08:20:29.834581 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.834584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.834635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.834743 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.835930 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.836443 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.837594 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.837814 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.837963 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.841280 (+ 3317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:59175: BlockingRecv error: recv got EOF from 127.12.13.3:59175 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:20:29.842150 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.832589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:59175)
0808 08:20:29.833396 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.833450 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.834359 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.834934 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.834941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.835279 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.835767 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.835780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.836579 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.836588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.837458 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.837467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.838006 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.838010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.838188 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.838677 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.838692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.841172 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":173,"threads_started":1}
W20260808 08:20:29.842857 26243 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:44269 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.13.62:44269: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
I20260808 08:20:29.843087 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.833474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.12.13.62:44269)
0808 08:20:29.833914 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.833917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.833927 (+ 10us) server_negotiation.cc:407] Connection header received
0808 08:20:29.833965 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.833967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.834005 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.834110 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.834884 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.835411 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.836282 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.836511 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.836621 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.838149 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:29.838155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:29.838169 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:29.838196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:29.840674 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:29.841138 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:29.841143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:29.841148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:29.841267 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:29.841609 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:29.841612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:29.841614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:29.842057 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:29.842261 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:29.842634 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 08:20:29.842892 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":85,"threads_started":1}
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:29.848857 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.843368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:56669)
0808 08:20:29.843497 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.843512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.843606 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.843864 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.843867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.844073 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.844185 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.844191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.844989 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.844992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.845731 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.845742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.846290 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.846293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.846368 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.846929 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.846942 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.848757 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 08:20:29.849017 26247 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.843418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:56669 (local address 127.12.13.62:44269)
0808 08:20:29.843517 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.843521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.843564 (+ 43us) server_negotiation.cc:407] Connection header received
0808 08:20:29.843709 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.843712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.843753 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.843842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.844304 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.844874 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.845847 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.846185 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.846271 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.848893 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:56669: BlockingRecv error: recv got EOF from 127.12.13.3:56669 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:29 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:29.854025 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:29.849463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:60747)
0808 08:20:29.849598 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:29.849610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:29.849673 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:29.849910 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:29.849915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:29.850042 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:29.850111 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.850116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.850793 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.850795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.851289 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:29.851296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.851695 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.851698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:29.851751 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.852213 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:29.852225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:29.853822 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 08:20:29.854267 26243 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:20:29.854270 26247 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:29.849561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:60747 (local address 127.12.13.62:44269)
0808 08:20:29.849693 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:29.849696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:29.849707 (+ 11us) server_negotiation.cc:407] Connection header received
0808 08:20:29.849739 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:29.849741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:29.849774 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:29.849840 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:29.850210 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.850701 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.851410 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:29.851605 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:29.851678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:29.854034 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:60747: BlockingRecv error: recv got EOF from 127.12.13.3:60747 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 08:20:30 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:30 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:30.455425 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:30.449414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:38177)
0808 08:20:30.449701 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:30.449718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:30.449837 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:30.450204 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:30.450208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:30.450479 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:30.450677 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:30.450687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.451772 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.451775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:30.452373 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:30.452382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.452789 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.452791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:30.452878 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:30.453383 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:30.453394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:30.455289 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
W20260808 08:20:30.455572 26251 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:30.449520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:38177 (local address 127.12.13.62:44269)
0808 08:20:30.449774 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:30.449778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:30.449793 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:20:30.449986 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:30.449989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:30.450055 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:30.450157 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:30.450937 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.451663 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.452477 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.452692 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.452766 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:30.455432 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:38177: BlockingRecv error: recv got EOF from 127.12.13.1:38177 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
Aug 08 08:20:30 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:30 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:30.661041 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:30.655168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:38891)
0808 08:20:30.655499 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:30.655513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:30.655657 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:30.655961 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:30.655964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:30.656186 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:30.656291 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:30.656297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.657105 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.657111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:30.657687 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:30.657695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.658256 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.658260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:30.658358 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:30.658896 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:30.658908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:30.660854 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":88,"threads_started":1}
W20260808 08:20:30.661147 26251 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:30.655239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:38891 (local address 127.12.13.62:44269)
0808 08:20:30.655426 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:30.655432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:30.655594 (+ 162us) server_negotiation.cc:407] Connection header received
0808 08:20:30.655761 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:30.655766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:30.655822 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:30.655933 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:30.656428 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.656986 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.657820 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.658133 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.658250 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:30.661025 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:38891: BlockingRecv error: recv got EOF from 127.12.13.2:38891 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 08:20:30 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:30 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:30.860715 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:30.854953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:41893)
0808 08:20:30.855273 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:30.855287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:30.855421 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:30.855709 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:30.855713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:30.855941 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:30.856058 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:30.856063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.856827 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.856831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:30.857424 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:30.857433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.857923 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.857927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:30.858022 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:30.858623 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:30.858636 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:30.860533 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
W20260808 08:20:30.860807 26251 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:30.855001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:41893 (local address 127.12.13.62:44269)
0808 08:20:30.855147 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:30.855152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:30.855364 (+ 212us) server_negotiation.cc:407] Connection header received
0808 08:20:30.855543 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:30.855547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:30.855596 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:30.855697 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:30.856177 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.856714 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.857556 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:30.857794 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:30.857933 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:30.860679 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:41893: BlockingRecv error: recv got EOF from 127.12.13.3:41893 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 08:20:31 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:31 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:31.465271 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:31.456390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:38829)
0808 08:20:31.456732 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:31.456748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:31.456884 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:31.457152 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:31.457154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:31.457357 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:31.457462 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:31.457468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.458363 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.458366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:31.458953 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:31.458962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.459388 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.459391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:31.459460 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:31.463088 (+ 3628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:31.463100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:31.465136 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
W20260808 08:20:31.465441 26256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:31.456446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:38829 (local address 127.12.13.62:44269)
0808 08:20:31.456719 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:31.456723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:31.456838 (+ 115us) server_negotiation.cc:407] Connection header received
0808 08:20:31.456977 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:31.456979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:31.457028 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:31.457110 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:31.457569 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.458252 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.459066 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.459282 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.459359 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:31.465289 (+ 5930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:38829: BlockingRecv error: recv got EOF from 127.12.13.1:38829 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
Aug 08 08:20:31 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:31 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:31.668488 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:31.661963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:37023)
0808 08:20:31.662244 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:31.662259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:31.662396 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:31.662741 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:31.662743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:31.662943 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:31.663044 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:31.663050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.664293 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.664299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:31.664990 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:31.665001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.665457 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.665460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:31.665521 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:31.666204 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:31.666226 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:31.668335 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260808 08:20:31.668623 26256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:31.662089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:37023 (local address 127.12.13.62:44269)
0808 08:20:31.662250 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:31.662255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:31.662328 (+ 73us) server_negotiation.cc:407] Connection header received
0808 08:20:31.662509 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:31.662512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:31.662570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:31.662657 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:31.663197 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.664116 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.665111 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.665337 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.665433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:31.668495 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:37023: BlockingRecv error: recv got EOF from 127.12.13.2:37023 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 08:20:31 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:31 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:31.867113 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:31.861594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:43131)
0808 08:20:31.861896 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:31.861915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:31.862018 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:31.862268 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:31.862270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:31.862484 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:31.862586 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:31.862591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.863377 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.863380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:31.863918 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:31.863925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.864387 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.864390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:31.864502 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:31.865097 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:31.865109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:31.866973 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
W20260808 08:20:31.867252 26256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:31.861770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:43131 (local address 127.12.13.62:44269)
0808 08:20:31.861965 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:31.861971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:31.861985 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:31.862107 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:31.862109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:31.862159 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:31.862227 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:31.862711 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.863258 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.864039 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:31.864260 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:31.864351 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:31.867124 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:43131: BlockingRecv error: recv got EOF from 127.12.13.3:43131 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 08:20:32 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:32 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:32.472186 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:32.466189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:39879)
0808 08:20:32.466492 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:32.466505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:32.466600 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:32.466891 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:32.466893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:32.467083 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:32.467199 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:32.467204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.468086 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.468089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:32.468735 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:32.468745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.469289 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.469292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:32.469357 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:32.469909 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:32.469926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:32.471996 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":138,"threads_started":1}
W20260808 08:20:32.472265 26267 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:32.466247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:39879 (local address 127.12.13.62:44269)
0808 08:20:32.466593 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:32.466597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:32.466611 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:32.466680 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:32.466682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:32.466728 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:32.466807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:32.467346 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.467951 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.468874 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.469120 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.469274 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:32.472132 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:39879: BlockingRecv error: recv got EOF from 127.12.13.1:39879 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":63,"threads_started":1}
Aug 08 08:20:32 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:32 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:32.675318 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:32.669544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:49033)
0808 08:20:32.669820 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:32.669844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:32.670019 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:32.670290 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:32.670292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:32.670511 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:32.670602 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:32.670608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.671371 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.671376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:32.671935 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:32.671943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.672419 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.672423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:32.672508 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:32.673031 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:32.673042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:32.675104 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260808 08:20:32.675329 26267 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:32.669625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:49033 (local address 127.12.13.62:44269)
0808 08:20:32.669816 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:32.669821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:32.669957 (+ 136us) server_negotiation.cc:407] Connection header received
0808 08:20:32.670119 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:32.670123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:32.670171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:32.670273 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:32.670728 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.671261 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.672066 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.672300 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.672406 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:32.675231 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:49033: BlockingRecv error: recv got EOF from 127.12.13.2:49033 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 08:20:32 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:32 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:32.874423 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:32.868065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:48115)
0808 08:20:32.868460 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:32.868474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:32.868627 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:32.868938 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:32.868940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:32.869160 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:32.869267 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:32.869273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.870133 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.870136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:32.870932 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:32.870947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.871573 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.871577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:32.871662 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:32.872208 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:32.872268 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:32.874200 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":96,"threads_started":1}
W20260808 08:20:32.874439 26267 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:32.868207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:48115 (local address 127.12.13.62:44269)
0808 08:20:32.868407 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:32.868412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:32.868559 (+ 147us) server_negotiation.cc:407] Connection header received
0808 08:20:32.868761 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:32.868769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:32.868819 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:32.868911 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:32.869395 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.870011 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.871174 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:32.871437 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:32.871556 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:32.874336 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:48115: BlockingRecv error: recv got EOF from 127.12.13.3:48115 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 08:20:33 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:33 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:33.479061 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:33.473019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:44021)
0808 08:20:33.473343 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:33.473360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:33.473468 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:33.473725 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:33.473727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:33.474043 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:33.474154 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:33.474160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.474898 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.474900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:33.475477 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:33.475485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.475949 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.475952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:33.476016 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:33.476576 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:33.476587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:33.478878 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
W20260808 08:20:33.479249 26276 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:33.473122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:44021 (local address 127.12.13.62:44269)
0808 08:20:33.473376 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:33.473380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:33.473416 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:20:33.473555 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:33.473557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:33.473603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:33.473683 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:33.474262 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.474794 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.475594 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.475817 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.475923 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:33.479106 (+ 3183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:44021: BlockingRecv error: recv got EOF from 127.12.13.1:44021 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
Aug 08 08:20:33 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:33 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:33.682308 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:33.676189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:49731)
0808 08:20:33.676520 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:33.676534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:33.676643 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:33.676898 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:33.676900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:33.677085 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:33.677184 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:33.677188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.678132 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.678136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:33.678840 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:33.678857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.679339 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.679342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:33.679405 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:33.679977 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:33.679989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:33.682185 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
W20260808 08:20:33.682642 26276 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:33.676355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:49731 (local address 127.12.13.62:44269)
0808 08:20:33.676541 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:33.676545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:33.676588 (+ 43us) server_negotiation.cc:407] Connection header received
0808 08:20:33.676730 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:33.676732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:33.676780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:33.676861 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:33.677283 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.677995 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.678978 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.679236 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.679321 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:33.682464 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:49731: BlockingRecv error: recv got EOF from 127.12.13.2:49731 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 08:20:33 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:33 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:33.881305 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:33.875261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:43835)
0808 08:20:33.875628 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:33.875642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:33.875786 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:33.876083 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:33.876086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:33.876308 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:33.876412 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:33.876418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.877197 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.877202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:33.877825 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:33.877842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.878416 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.878419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:33.878510 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:33.879140 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:33.879152 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:33.881118 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":95,"threads_started":1}
W20260808 08:20:33.881400 26276 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:33.875400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:43835 (local address 127.12.13.62:44269)
0808 08:20:33.875568 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:33.875573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:33.875719 (+ 146us) server_negotiation.cc:407] Connection header received
0808 08:20:33.875898 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:33.875902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:33.875961 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:33.876058 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:33.876535 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.877075 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.877999 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:33.878231 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:33.878408 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:33.881260 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:43835: BlockingRecv error: recv got EOF from 127.12.13.3:43835 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 08:20:34 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:34 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:34.485656 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:34.480022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:39395)
0808 08:20:34.480392 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:34.480406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:34.480503 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:34.480756 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:34.480758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:34.480965 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:34.481069 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:34.481074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.481901 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.481903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:34.482476 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:34.482484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.482972 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.482975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:34.483035 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:34.483550 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:34.483561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:34.485522 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":100,"threads_started":1}
W20260808 08:20:34.485831 26283 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:34.480062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:39395 (local address 127.12.13.62:44269)
0808 08:20:34.480370 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:34.480374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:34.480461 (+ 87us) server_negotiation.cc:407] Connection header received
0808 08:20:34.480573 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:34.480575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:34.480621 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:34.480719 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:34.481221 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.481763 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.482618 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.482844 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.482953 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:34.485684 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:39395: BlockingRecv error: recv got EOF from 127.12.13.1:39395 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":74,"threads_started":1}
Aug 08 08:20:34 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:34 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:34.694223 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:34.683275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:40873)
0808 08:20:34.683572 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:34.683585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:34.683681 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:34.683931 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:34.683933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:34.684125 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:34.684224 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:34.684229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.684981 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.684983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:34.685558 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:34.685566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.686171 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.686174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:34.686248 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:34.691635 (+ 5387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:34.691649 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:34.694039 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
W20260808 08:20:34.694296 26283 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:34.683414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:40873 (local address 127.12.13.62:44269)
0808 08:20:34.683577 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:34.683581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:34.683656 (+ 75us) server_negotiation.cc:407] Connection header received
0808 08:20:34.683764 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:34.683767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:34.683814 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:34.683894 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:34.684328 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.684876 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.685680 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.685949 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.686056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:34.694183 (+ 8127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:40873: BlockingRecv error: recv got EOF from 127.12.13.2:40873 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 08:20:34 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:34 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:34.890851 26285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:34.882565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:33861)
0808 08:20:34.882852 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:34.882869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:34.882978 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:34.883245 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:34.883249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:34.883548 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:34.883712 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:34.883722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.884862 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.884868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:34.885992 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:34.886005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.886787 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.886794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:34.886909 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:34.887851 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:34.887870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:34.890622 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
W20260808 08:20:34.891319 26283 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:34.882644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:33861 (local address 127.12.13.62:44269)
0808 08:20:34.882818 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:34.882825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:34.882954 (+ 129us) server_negotiation.cc:407] Connection header received
0808 08:20:34.883050 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:34.883053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:34.883101 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:34.883173 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:34.883860 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.884721 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.886164 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:34.886464 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:34.886581 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:34.890835 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:33861: BlockingRecv error: recv got EOF from 127.12.13.3:33861 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 08:20:35 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:35 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:35.493077 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:35.486771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:35745)
0808 08:20:35.487231 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:35.487247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:35.487368 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:35.487619 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:35.487621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:35.487815 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:35.487918 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:35.487922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.488796 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.488799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:35.489394 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:35.489404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.490049 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.490051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:35.490153 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:35.490728 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:35.490740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:35.492951 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":138,"threads_started":1}
W20260808 08:20:35.493297 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:35.486873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:35745 (local address 127.12.13.62:44269)
0808 08:20:35.487140 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:35.487146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:35.487397 (+ 251us) server_negotiation.cc:407] Connection header received
0808 08:20:35.487447 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:35.487449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:35.487499 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:35.487583 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:35.488038 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.488669 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.489539 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.489899 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.490023 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:35.493111 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:35745: BlockingRecv error: recv got EOF from 127.12.13.1:35745 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
Aug 08 08:20:35 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:35 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:35.701385 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:35.695160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:48033)
0808 08:20:35.695465 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:35.695479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:35.695573 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:35.695887 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:35.695890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:35.696084 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:35.696184 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:35.696189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.696972 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.696974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:35.697518 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:35.697525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.698035 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.698037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:35.698099 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:35.698641 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:35.698653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:35.701182 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
W20260808 08:20:35.701526 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:35.695313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:48033 (local address 127.12.13.62:44269)
0808 08:20:35.695505 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:35.695509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:35.695528 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:20:35.695706 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:35.695709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:35.695764 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:35.695869 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:35.696305 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.696866 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.697644 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.697919 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.698007 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:35.701361 (+ 3354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:48033: BlockingRecv error: recv got EOF from 127.12.13.2:48033 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 08:20:35 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:35 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:35.897560 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:35.891921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:40493)
0808 08:20:35.892214 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:35.892232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:35.892362 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:35.892662 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:35.892664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:35.892872 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:35.892974 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:35.892980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.893745 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.893747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:35.894372 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:35.894381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.894799 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.894802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:35.894868 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:35.895467 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:35.895480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:35.897383 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
W20260808 08:20:35.897605 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:35.891993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:40493 (local address 127.12.13.62:44269)
0808 08:20:35.892156 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:35.892160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:35.892315 (+ 155us) server_negotiation.cc:407] Connection header received
0808 08:20:35.892457 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:35.892460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:35.892508 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:35.892576 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:35.893101 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.893635 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.894481 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:35.894697 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:35.894777 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:35.897502 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:40493: BlockingRecv error: recv got EOF from 127.12.13.3:40493 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 08:20:36 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:36 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:36.504769 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:36.493980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:43485)
0808 08:20:36.494426 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:36.494440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:36.494558 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:36.494851 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:36.494853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:36.495045 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:36.495143 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:36.495148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.495949 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.495952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:36.496528 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:36.496538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.496999 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.497001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:36.497062 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:36.502605 (+ 5543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:36.502629 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:36.504642 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":102,"threads_started":1}
W20260808 08:20:36.505123 26292 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:36.494131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:43485 (local address 127.12.13.62:44269)
0808 08:20:36.494456 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:36.494460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:36.494525 (+ 65us) server_negotiation.cc:407] Connection header received
0808 08:20:36.494663 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:36.494666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:36.494717 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:36.494805 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:36.495259 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.495813 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.496638 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.496892 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.496973 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:36.504948 (+ 7975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:43485: BlockingRecv error: recv got EOF from 127.12.13.1:43485 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
Aug 08 08:20:36 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:36 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:36.708496 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:36.702308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:53941)
0808 08:20:36.702655 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:36.702670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:36.702792 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:36.703068 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:36.703070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:36.703263 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:36.703362 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:36.703366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.704142 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.704144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:36.704685 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:36.704693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.705194 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.705197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:36.705263 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:36.705997 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:36.706012 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:36.708376 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
W20260808 08:20:36.708645 26292 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:36.702387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:53941 (local address 127.12.13.62:44269)
0808 08:20:36.702561 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:36.702566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:36.702755 (+ 189us) server_negotiation.cc:407] Connection header received
0808 08:20:36.702897 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:36.702900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:36.702950 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:36.703024 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:36.703466 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.704040 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.704838 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.705089 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.705166 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:36.708524 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:53941: BlockingRecv error: recv got EOF from 127.12.13.2:53941 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 08:20:36 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:36 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:36.904467 26294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:36.898455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:57745)
0808 08:20:36.898806 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:36.898818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:36.898995 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:36.899306 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:36.899309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:36.899558 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:36.899651 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:36.899657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.900542 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.900547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:36.901102 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:36.901109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.901614 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.901618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:36.901703 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:36.902331 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:36.902343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:36.904274 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":92,"threads_started":1}
W20260808 08:20:36.904543 26292 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:36.898529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:57745 (local address 127.12.13.62:44269)
0808 08:20:36.898662 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:36.898667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:36.898950 (+ 283us) server_negotiation.cc:407] Connection header received
0808 08:20:36.899103 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:36.899108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:36.899174 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:36.899280 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:36.899774 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.900410 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.901226 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:36.901488 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:36.901600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:36.904427 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:57745: BlockingRecv error: recv got EOF from 127.12.13.3:57745 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 08 08:20:37 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:37 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:37.511530 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:37.505643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:58869)
0808 08:20:37.506031 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:37.506046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:37.506139 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:37.506425 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:37.506427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:37.506684 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:37.506797 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:37.506802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.507596 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.507598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:37.508169 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:37.508177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.508653 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.508656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:37.508725 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:37.509268 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:37.509283 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:37.511398 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":127,"threads_started":1}
W20260808 08:20:37.511734 26297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:37.505838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:58869 (local address 127.12.13.62:44269)
0808 08:20:37.506129 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:37.506133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:37.506147 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:37.506234 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:37.506239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:37.506298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:37.506385 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:37.506916 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.507490 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.508272 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.508530 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.508612 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:37.511550 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:58869: BlockingRecv error: recv got EOF from 127.12.13.1:58869 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
Aug 08 08:20:37 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:37 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:37.715840 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:37.709414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:53609)
0808 08:20:37.709891 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:37.709912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:37.710048 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:37.710362 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:37.710365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:37.710592 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:37.710694 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:37.710700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.711637 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.711643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:37.712216 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:37.712225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.712699 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.712702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:37.712786 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:37.713470 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:37.713484 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:37.715681 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":115,"threads_started":1}
W20260808 08:20:37.716001 26297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:37.709546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:53609 (local address 127.12.13.62:44269)
0808 08:20:37.709837 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:37.709842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:37.709989 (+ 147us) server_negotiation.cc:407] Connection header received
0808 08:20:37.710161 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:37.710165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:37.710223 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:37.710331 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:37.710830 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.711520 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.712342 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.712575 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.712691 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:37.715868 (+ 3177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:53609: BlockingRecv error: recv got EOF from 127.12.13.2:53609 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Aug 08 08:20:37 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:37 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:37.911712 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:37.905519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:54251)
0808 08:20:37.905932 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:37.905944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:37.906086 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:37.906394 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:37.906397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:37.906616 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:37.906710 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:37.906716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.907533 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.907539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:37.908092 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:37.908100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.908631 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.908634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:37.908719 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:37.909375 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:37.909391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:37.911591 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":107,"threads_started":1}
W20260808 08:20:37.911913 26297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:37.905596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:54251 (local address 127.12.13.62:44269)
0808 08:20:37.905762 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:37.905769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:37.906033 (+ 264us) server_negotiation.cc:407] Connection header received
0808 08:20:37.906220 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:37.906223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:37.906273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:37.906363 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:37.906838 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.907420 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.908263 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:37.908498 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:37.908610 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:37.911742 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:54251: BlockingRecv error: recv got EOF from 127.12.13.3:54251 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 08:20:38 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:38 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:38.518020 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:38.512380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:41069)
0808 08:20:38.512682 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:38.512695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:38.512791 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:38.513075 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:38.513077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:38.513265 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:38.513372 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:38.513377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.514220 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.514223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:38.514823 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:38.514831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.515269 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.515272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:38.515332 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:38.515872 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:38.515884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:38.517825 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
W20260808 08:20:38.518210 26304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:38.512471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:41069 (local address 127.12.13.62:44269)
0808 08:20:38.512717 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:38.512721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:38.512738 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:20:38.512902 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:38.512904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:38.512954 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:38.513034 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:38.513481 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.514111 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.514943 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.515166 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.515245 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:38.518057 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:41069: BlockingRecv error: recv got EOF from 127.12.13.1:41069 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
Aug 08 08:20:38 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:38 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:38.722337 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:38.716795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:43735)
0808 08:20:38.717084 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:38.717097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:38.717203 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:38.717450 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:38.717453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:38.717640 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:38.717742 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:38.717748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.718625 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.718627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:38.719191 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:38.719199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.719638 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.719641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:38.719701 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:38.720242 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:38.720253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:38.722215 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
W20260808 08:20:38.722539 26304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:38.716880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:43735 (local address 127.12.13.62:44269)
0808 08:20:38.717055 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:38.717059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:38.717150 (+ 91us) server_negotiation.cc:407] Connection header received
0808 08:20:38.717286 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:38.717288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:38.717370 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:38.717435 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:38.717857 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.718492 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.719299 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.719536 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.719617 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:38.722403 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:43735: BlockingRecv error: recv got EOF from 127.12.13.2:43735 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 08:20:38 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:38 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:38.919014 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:38.912615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:55475)
0808 08:20:38.913041 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:38.913058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:38.913225 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:38.913533 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:38.913536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:38.913784 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:38.913918 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:38.913924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.914777 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.914782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:38.915397 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:38.915407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.915918 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.915922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:38.916021 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:38.916610 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:38.916622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:38.918891 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":150,"threads_started":1}
W20260808 08:20:38.919239 26304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:38.912797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:55475 (local address 127.12.13.62:44269)
0808 08:20:38.913008 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:38.913014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:38.913154 (+ 140us) server_negotiation.cc:407] Connection header received
0808 08:20:38.913339 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:38.913343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:38.913401 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:38.913506 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:38.914057 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.914655 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.915530 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:38.915755 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:38.915890 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:38.919084 (+ 3194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:55475: BlockingRecv error: recv got EOF from 127.12.13.3:55475 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 08 08:20:39 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:39 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:39.525254 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:39.518984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:48313)
0808 08:20:39.519305 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:39.519319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:39.519419 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:39.519673 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:39.519675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:39.519891 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:39.519995 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:39.520000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.520809 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.520812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:39.521389 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:39.521399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.522125 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.522129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:39.522323 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:39.523018 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:39.523032 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:39.525129 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":134,"threads_started":1}
W20260808 08:20:39.525429 26309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:39.519094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:48313 (local address 127.12.13.62:44269)
0808 08:20:39.519416 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:39.519420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:39.519434 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:20:39.519498 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:39.519500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:39.519544 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:39.519629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:39.520110 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.520664 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.521560 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.521988 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.522119 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:39.525287 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:48313: BlockingRecv error: recv got EOF from 127.12.13.1:48313 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
Aug 08 08:20:39 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:39 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 08:20:39.729378 26309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:39.723318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:53139 (local address 127.12.13.62:44269)
0808 08:20:39.723514 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:39.723518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:39.723637 (+ 119us) server_negotiation.cc:407] Connection header received
0808 08:20:39.723771 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:39.723773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:39.723819 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:39.723899 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:39.724350 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.724887 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.725703 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.726023 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.726105 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:39.729277 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:53139: BlockingRecv error: recv got EOF from 127.12.13.2:53139 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 08:20:39.729387 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:39.723223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:53139)
0808 08:20:39.723566 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:39.723580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:39.723686 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:39.723943 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:39.723945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:39.724139 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:39.724241 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:39.724246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.724993 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.724996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:39.725590 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:39.725599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.726144 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.726148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:39.726248 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:39.726993 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:39.727009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:39.729147 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
Aug 08 08:20:39 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:39 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:39.925572 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:39.919923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:54629)
0808 08:20:39.920283 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:39.920298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:39.920407 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:39.920657 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:39.920660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:39.920901 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:39.921008 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:39.921014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.921835 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.921838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:39.922424 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:39.922433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.922882 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.922885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:39.922950 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:39.923473 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:39.923485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:39.925395 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
W20260808 08:20:39.925637 26309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:39.920129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:54629 (local address 127.12.13.62:44269)
0808 08:20:39.920287 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:39.920291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:39.920390 (+ 99us) server_negotiation.cc:407] Connection header received
0808 08:20:39.920491 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:39.920493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:39.920542 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:39.920610 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:39.921112 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.921705 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.922528 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:39.922742 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:39.922823 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:39.925522 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:54629: BlockingRecv error: recv got EOF from 127.12.13.3:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 08:20:40 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:40 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:40.531816 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:40.526159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:59335)
0808 08:20:40.526466 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:40.526483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:40.526596 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:40.526906 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:40.526908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:40.527100 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:40.527205 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:40.527210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.528015 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.528017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:40.528560 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:40.528567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.528995 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.528998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:40.529060 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:40.529589 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:40.529601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:40.531626 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
W20260808 08:20:40.531929 26314 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:40.526350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59335 (local address 127.12.13.62:44269)
0808 08:20:40.526629 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:40.526633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:40.526656 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:20:40.526701 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:40.526703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:40.526751 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:40.526849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:40.527346 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.527908 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.528661 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.528890 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.528968 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:40.531766 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:59335: BlockingRecv error: recv got EOF from 127.12.13.1:59335 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
Aug 08 08:20:40 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:40 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:40.735841 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:40.730200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:36975)
0808 08:20:40.730510 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:40.730525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:40.730639 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:40.730879 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:40.730881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:40.731070 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:40.731171 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:40.731176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.731894 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.731896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:40.732440 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:40.732448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.732867 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.732869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:40.732929 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:40.733473 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:40.733485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:40.735707 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
W20260808 08:20:40.735977 26314 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:40.730262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:36975 (local address 127.12.13.62:44269)
0808 08:20:40.730424 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:40.730429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:40.730605 (+ 176us) server_negotiation.cc:407] Connection header received
0808 08:20:40.730712 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:40.730714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:40.730763 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:40.730833 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:40.731269 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.731784 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.732540 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.732749 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.732823 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:40.735858 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:36975: BlockingRecv error: recv got EOF from 127.12.13.2:36975 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 08:20:40 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:40 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:40.932153 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:40.926571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:36199)
0808 08:20:40.926870 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:40.926893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:40.927001 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:40.927244 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:40.927247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:40.927479 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:40.927582 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:40.927588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.928362 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.928365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:40.928914 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:40.928923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.929358 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.929361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:40.929424 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:40.930053 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:40.930066 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:40.932025 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
W20260808 08:20:40.932385 26314 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:40.926637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:36199 (local address 127.12.13.62:44269)
0808 08:20:40.926794 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:40.926798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:40.926962 (+ 164us) server_negotiation.cc:407] Connection header received
0808 08:20:40.927081 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:40.927083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:40.927134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:40.927203 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:40.927695 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.928227 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.929016 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:40.929233 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:40.929309 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:40.932184 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:36199: BlockingRecv error: recv got EOF from 127.12.13.3:36199 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 08:20:41 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:41 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:41.543200 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:41.532925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:58761)
0808 08:20:41.533237 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:41.533255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:41.533381 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:41.533638 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:41.533640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:41.533831 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:41.533984 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:41.533989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.534786 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.534789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:41.535339 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:41.535346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.535965 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.535970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:41.536075 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:41.540881 (+ 4806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:41.540896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:41.543012 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
W20260808 08:20:41.543306 26318 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:41.533098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:58761 (local address 127.12.13.62:44269)
0808 08:20:41.533396 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:41.533400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:41.533418 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:20:41.533463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:41.533465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:41.533512 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:41.533596 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:41.534108 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.534683 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.535486 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.535830 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.535945 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:41.543165 (+ 7220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:58761: BlockingRecv error: recv got EOF from 127.12.13.1:58761 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
Aug 08 08:20:41 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:41 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:41.742766 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:41.736860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:45209)
0808 08:20:41.737176 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:41.737191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:41.737293 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:41.737579 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:41.737582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:41.737780 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:41.737944 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:41.737951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.738855 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.738858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:41.739603 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:41.739613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.740160 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.740163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:41.740234 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:41.740772 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:41.740784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:41.742642 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
W20260808 08:20:41.742908 26318 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:41.737101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:45209 (local address 127.12.13.62:44269)
0808 08:20:41.737253 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:41.737257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:41.737273 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:20:41.737398 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:41.737400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:41.737448 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:41.737533 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:41.738050 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.738739 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.739723 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.740023 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.740106 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:41.742782 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:45209: BlockingRecv error: recv got EOF from 127.12.13.2:45209 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 08:20:41 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:41 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:41.939002 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:41.933003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:45907)
0808 08:20:41.933380 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:41.933394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:41.933544 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:41.933845 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:41.933848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:41.934111 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:41.934219 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:41.934225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.935077 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.935082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:41.935640 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:41.935648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.936152 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.936156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:41.936243 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:41.936838 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:41.936852 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:41.938870 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":93,"threads_started":1}
W20260808 08:20:41.939287 26318 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:41.933142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:45907 (local address 127.12.13.62:44269)
0808 08:20:41.933318 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:41.933323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:41.933481 (+ 158us) server_negotiation.cc:407] Connection header received
0808 08:20:41.933656 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:41.933660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:41.933713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:41.933815 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:41.934355 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.934949 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.935770 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:41.936020 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:41.936134 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:41.939018 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:45907: BlockingRecv error: recv got EOF from 127.12.13.3:45907 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 08:20:42 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:42 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:42.550338 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:42.544186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:35859)
0808 08:20:42.544642 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:42.544656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:42.544766 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:42.545088 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:42.545090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:42.545289 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:42.545400 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:42.545406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.546372 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.546378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:42.547037 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:42.547047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.547524 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.547527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:42.547586 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:42.548175 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:42.548187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:42.550141 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":211,"threads_started":1}
W20260808 08:20:42.550619 26324 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:42.544335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:35859 (local address 127.12.13.62:44269)
0808 08:20:42.544643 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:42.544647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:42.544732 (+ 85us) server_negotiation.cc:407] Connection header received
0808 08:20:42.544859 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:42.544861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:42.544907 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:42.544999 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:42.545542 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.546197 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.547150 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.547415 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.547504 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:42.550345 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:35859: BlockingRecv error: recv got EOF from 127.12.13.1:35859 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
Aug 08 08:20:42 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:42 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:42.749679 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:42.743617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:40831)
0808 08:20:42.743943 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:42.743957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:42.744077 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:42.744332 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:42.744334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:42.744521 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:42.744619 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:42.744624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.745461 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.745464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:42.746200 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:42.746209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.746655 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.746658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:42.746722 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:42.747514 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:42.747527 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:42.749494 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
W20260808 08:20:42.749791 26324 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:42.743793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:40831 (local address 127.12.13.62:44269)
0808 08:20:42.743959 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:42.743963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:42.744027 (+ 64us) server_negotiation.cc:407] Connection header received
0808 08:20:42.744171 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:42.744173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:42.744224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:42.744293 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:42.744724 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.745325 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.746308 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.746538 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.746629 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:42.749629 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:40831: BlockingRecv error: recv got EOF from 127.12.13.2:40831 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 08:20:42 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:42 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:42.945282 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:42.939943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:51309)
0808 08:20:42.940296 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:42.940310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:42.940423 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:42.940652 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:42.940655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:42.940874 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:42.940976 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:42.940982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.941714 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.941716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:42.942308 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:42.942316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.942719 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.942722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:42.942782 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:42.943321 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:42.943335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:42.945152 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":157,"threads_started":1}
W20260808 08:20:42.945418 26324 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:42.939959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:51309 (local address 127.12.13.62:44269)
0808 08:20:42.940104 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:42.940115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:42.940377 (+ 262us) server_negotiation.cc:407] Connection header received
0808 08:20:42.940496 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:42.940498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:42.940545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:42.940611 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:42.941092 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.941620 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.942411 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:42.942621 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:42.942699 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:42.945292 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:51309: BlockingRecv error: recv got EOF from 127.12.13.3:51309 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 08:20:43 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:43 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:43.557775 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:43.551256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:48707)
0808 08:20:43.551556 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:43.551570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:43.551670 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:43.552005 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:43.552008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:43.552323 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:43.552447 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:43.552455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.553278 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.553281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:43.553973 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:43.553990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.554483 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.554486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:43.554569 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:43.555262 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:43.555281 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:43.557646 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
W20260808 08:20:43.558059 26332 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:43.551375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:48707 (local address 127.12.13.62:44269)
0808 08:20:43.551709 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:43.551713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:43.551733 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:20:43.551775 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:43.551777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:43.551821 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:43.551911 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:43.552595 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.553147 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.554106 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.554347 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.554433 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:43.557869 (+ 3436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:48707: BlockingRecv error: recv got EOF from 127.12.13.1:48707 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
Aug 08 08:20:43 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:43 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:43.756724 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:43.750772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:55305)
0808 08:20:43.751100 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:43.751117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:43.751253 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:43.751570 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:43.751573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:43.751800 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:43.751935 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:43.751942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.752716 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.752719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:43.753365 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:43.753375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.753828 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.753831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:43.753935 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:43.754469 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:43.754484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:43.756595 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":132,"threads_started":1}
W20260808 08:20:43.756901 26332 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:43.750845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:55305 (local address 127.12.13.62:44269)
0808 08:20:43.751020 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:43.751025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:43.751201 (+ 176us) server_negotiation.cc:407] Connection header received
0808 08:20:43.751367 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:43.751369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:43.751419 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:43.751492 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:43.752052 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.752587 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.753481 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.753693 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.753770 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:43.756736 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:55305: BlockingRecv error: recv got EOF from 127.12.13.2:55305 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 08:20:43 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:43 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.3@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:43.951923 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:43.946205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.3:53273)
0808 08:20:43.946487 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:43.946500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:43.946603 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:43.946934 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:43.946938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:43.947141 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:43.947244 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:43.947250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.948124 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.948127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:43.948673 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:43.948681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.949109 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.949112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:43.949172 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:43.949749 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:43.949762 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:43.951764 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
W20260808 08:20:43.952131 26332 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:43.946295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:53273 (local address 127.12.13.62:44269)
0808 08:20:43.946462 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:43.946466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:43.946551 (+ 85us) server_negotiation.cc:407] Connection header received
0808 08:20:43.946693 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:43.946696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:43.946758 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:43.946858 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:43.947389 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.948022 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.948779 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:43.949010 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:43.949090 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:43.951964 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.3:53273: BlockingRecv error: recv got EOF from 127.12.13.3:53273 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 08 08:20:44 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:44 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.1@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:44.565187 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:44.558965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.1:47155)
0808 08:20:44.559318 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:44.559336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:44.559444 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:44.559990 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:44.559993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:44.560254 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:44.560386 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:44.560397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.561246 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.561249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:44.561798 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:44.561806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.562284 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.562286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:44.562351 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:44.562878 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:44.562892 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:44.565010 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":134,"threads_started":1}
W20260808 08:20:44.565263 26338 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:44.559364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:47155 (local address 127.12.13.62:44269)
0808 08:20:44.559633 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:44.559640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:44.559666 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:20:44.559737 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:44.559740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:44.559798 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:44.559912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:44.560546 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.561140 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.561947 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.562177 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.562259 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:44.565137 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.1:47155: BlockingRecv error: recv got EOF from 127.12.13.1:47155 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
Aug 08 08:20:44 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 08:20:44 dist-test-slave-6ntk krb5kdc[24712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.13.2@KRBTEST.COM for oryx/127.12.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 08:20:44.763386 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:44.757832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44269 (local address 127.12.13.2:34643)
0808 08:20:44.758171 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:44.758185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:44.758284 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:44.758529 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:44.758531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:44.758718 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:44.758819 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:44.758824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.759558 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.759561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:44.760161 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:44.760170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.760635 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.760638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:44.760698 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:44.761302 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:44.761315 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:44.763239 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.13.62:44269: Server oryx/127.12.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
W20260808 08:20:44.763578 26338 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:44.757970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:34643 (local address 127.12.13.62:44269)
0808 08:20:44.758151 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:44.758161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:44.758257 (+ 96us) server_negotiation.cc:407] Connection header received
0808 08:20:44.758363 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:44.758366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:44.758415 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:44.758489 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:44.758927 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.759456 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.760289 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:44.760529 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:44.760617 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:44.763403 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.13.2:34643: BlockingRecv error: recv got EOF from 127.12.13.2:34643 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 08:20:44.834692 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 25653
I20260808 08:20:44.859090 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 25864
I20260808 08:20:44.872128 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 26057
I20260808 08:20:44.887342 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 25553
2026-08-08T08:20:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18480 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T08:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:44Z Disabled control of system clock
I20260808 08:20:44.939476 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:40103
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 08:20:45.061105 26345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:45.061480 26345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:45.061522 26345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:45.065557 26345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:45.065619 26345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:45.065634 26345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:45.065651 26345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:45.065667 26345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:45.070447 26345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:40103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:45.071555 26345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:45.072962 26345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:45.082206 26352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.082206 26351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.082239 26354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:45.082530 26345 server_base.cc:1034] running on GCE node
I20260808 08:20:45.083278 26345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:45.084640 26345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:45.085927 26345 hybrid_clock.cc:648] HybridClock initialized: now 1786177245085837 us; error 60 us; skew 500 ppm
I20260808 08:20:45.087983 26345 webserver.cc:460] Webserver started at http://127.12.13.62:42425/ using document root <none> and password file <none>
I20260808 08:20:45.088640 26345 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:45.088742 26345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:45.088999 26345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:45.091595 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "4504b5e75c9d4f4080bdba6fbe510126"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "42c4e81f2a98e65312a4fcdcb78b602d"
server_key_iv: "cf6d5e04cb86151680b8ce4dadc5229a"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.092274 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "4504b5e75c9d4f4080bdba6fbe510126"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "42c4e81f2a98e65312a4fcdcb78b602d"
server_key_iv: "cf6d5e04cb86151680b8ce4dadc5229a"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.096325 26345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:45.099027 26361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.100334 26345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:45.100507 26345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "4504b5e75c9d4f4080bdba6fbe510126"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "42c4e81f2a98e65312a4fcdcb78b602d"
server_key_iv: "cf6d5e04cb86151680b8ce4dadc5229a"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.100633 26345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:45.111195 26345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:45.111923 26345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:45.112125 26345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:45.119825 26345 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:40103
I20260808 08:20:45.119833 26439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:40103 every 8 connection(s)
I20260808 08:20:45.120810 26345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:45.124311 26440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:45.124627 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 26345
I20260808 08:20:45.124763 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:45.125079 12340 external_mini_cluster.cc:1428] Setting key 68eec23500b2cc79388ed6f69da14a07
I20260808 08:20:45.131634 26440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126: Bootstrap starting.
I20260808 08:20:45.134473 26440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:45.135375 26440 log.cc:824] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:45.135852 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.126640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.12.13.62:40103)
0808 08:20:45.127377 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:45.127401 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:45.127447 (+ 46us) server_negotiation.cc:407] Connection header received
0808 08:20:45.128301 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:45.128326 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:45.128852 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:45.129229 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:45.129696 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.131169 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.132116 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.132788 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.132972 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.133753 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:45.133759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:45.133776 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:45.134308 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:45.134397 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:45.134430 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:45.134540 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:45.134860 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 08:20:45.135078 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":127,"threads_started":1}
I20260808 08:20:45.137555 26440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126: No bootstrap required, opened a new log
I20260808 08:20:45.141960 26440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4504b5e75c9d4f4080bdba6fbe510126" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40103 } }
I20260808 08:20:45.142203 26440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:45.142241 26440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4504b5e75c9d4f4080bdba6fbe510126, State: Initialized, Role: FOLLOWER
I20260808 08:20:45.142675 26440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4504b5e75c9d4f4080bdba6fbe510126" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40103 } }
I20260808 08:20:45.142803 26440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:45.142858 26440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:45.142928 26440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:45.143822 26440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4504b5e75c9d4f4080bdba6fbe510126" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40103 } }
I20260808 08:20:45.144140 26440 leader_election.cc:302] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4504b5e75c9d4f4080bdba6fbe510126; no voters:
I20260808 08:20:45.144413 26440 leader_election.cc:288] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:45.144541 26447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:45.144783 26447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [term 1 LEADER]: Becoming Leader. State: Replica: 4504b5e75c9d4f4080bdba6fbe510126, State: Running, Role: LEADER
I20260808 08:20:45.145151 26447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4504b5e75c9d4f4080bdba6fbe510126" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40103 } }
I20260808 08:20:45.145818 26440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:45.147179 26449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4504b5e75c9d4f4080bdba6fbe510126. Latest consensus state: current_term: 1 leader_uuid: "4504b5e75c9d4f4080bdba6fbe510126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4504b5e75c9d4f4080bdba6fbe510126" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40103 } } }
I20260808 08:20:45.147240 26448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4504b5e75c9d4f4080bdba6fbe510126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4504b5e75c9d4f4080bdba6fbe510126" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40103 } } }
I20260808 08:20:45.147439 26449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:45.147471 26448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126 [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:45.147895 26458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:45.151000 26458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:45.156904 26458 catalog_manager.cc:1347] Generated new cluster ID: 034ff22c03904c5bb2e7bf67dc26175e
I20260808 08:20:45.156991 26458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:45.170454 26458 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:45.171671 26458 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:45.176591 26458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4504b5e75c9d4f4080bdba6fbe510126: Generated new TSK 0
I20260808 08:20:45.177309 26458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:45.181811 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40103
--builtin_ntp_servers=127.12.13.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 08:20:45.308820 26475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:45.309132 26475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:45.309170 26475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:45.313295 26475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:45.313357 26475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:45.313410 26475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:45.318197 26475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:45.319289 26475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:45.320724 26475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:45.331053 26487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.331053 26482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.331079 26484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:45.331416 26475 server_base.cc:1034] running on GCE node
I20260808 08:20:45.331884 26475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:45.332515 26475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:45.333683 26475 hybrid_clock.cc:648] HybridClock initialized: now 1786177245333661 us; error 35 us; skew 500 ppm
I20260808 08:20:45.335548 26475 webserver.cc:460] Webserver started at http://127.12.13.1:43357/ using document root <none> and password file <none>
I20260808 08:20:45.336170 26475 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:45.336236 26475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:45.336418 26475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:45.338542 26475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "c79cfcb7d9a84cb89270276244f25415"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "553f94336c7b13092dcf8e32a9fb0dec"
server_key_iv: "25bba0abe8d92421fc939e5d0929bbd6"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.339059 26475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "c79cfcb7d9a84cb89270276244f25415"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "553f94336c7b13092dcf8e32a9fb0dec"
server_key_iv: "25bba0abe8d92421fc939e5d0929bbd6"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.342792 26475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:45.345412 26493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.346871 26475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:20:45.347007 26475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "c79cfcb7d9a84cb89270276244f25415"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "553f94336c7b13092dcf8e32a9fb0dec"
server_key_iv: "25bba0abe8d92421fc939e5d0929bbd6"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.347141 26475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:45.377419 26475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:45.378248 26475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:45.378476 26475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:45.379164 26475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:45.380199 26475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:45.380249 26475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.380321 26475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:45.380362 26475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.389760 26475 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:32903
I20260808 08:20:45.389783 26656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:32903 every 8 connection(s)
I20260808 08:20:45.390797 26475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:45.391431 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 26475
I20260808 08:20:45.391569 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:45.391860 12340 external_mini_cluster.cc:1428] Setting key 7f15be194651392307e5a41883d127c6
I20260808 08:20:45.394269 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40103
--builtin_ntp_servers=127.12.13.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 08:20:45.402441 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.392762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40803 (local address 127.12.13.62:40103)
0808 08:20:45.392920 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:45.392925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:45.394308 (+ 1383us) server_negotiation.cc:407] Connection header received
0808 08:20:45.396605 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:45.396612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:45.396689 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:45.396796 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:45.397790 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.398430 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.399794 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.400229 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.400381 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.401723 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:45.401728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:45.401734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:45.401756 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:45.401784 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:45.401799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:45.401921 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:45.402106 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 08:20:45.402199 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 08:20:45.403285 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.393220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:40103 (local address 127.12.13.1:40803)
0808 08:20:45.394143 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:45.394207 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:45.396287 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:45.396971 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:45.396979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:45.397073 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:45.397627 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:45.397641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.398584 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.398587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:45.399640 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:45.399653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.400433 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.400439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:45.400636 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.401411 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:45.401413 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:45.401461 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:45.401535 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:45.401945 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:45.401953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:45.401958 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 08:20:45.402067 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 08:20:45.402349 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":157,"threads_started":1}
I20260808 08:20:45.405074 26657 heartbeater.cc:347] Connected to a master server at 127.12.13.62:40103
I20260808 08:20:45.405393 26657 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:45.406167 26657 heartbeater.cc:511] Master 127.12.13.62:40103 requested a full tablet report, sending...
I20260808 08:20:45.408231 26389 ts_manager.cc:194] Registered new tserver with Master: c79cfcb7d9a84cb89270276244f25415 (127.12.13.1:32903)
I20260808 08:20:45.409991 26389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.1:40803
W20260808 08:20:45.517694 26660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:45.518028 26660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:45.518071 26660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:45.522063 26660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:45.522130 26660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:45.522341 26660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:45.527916 26660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:45.529074 26660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:45.530620 26660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:45.541134 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.541152 26666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.541134 26668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:45.541510 26660 server_base.cc:1034] running on GCE node
I20260808 08:20:45.542235 26660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:45.542960 26660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:45.544154 26660 hybrid_clock.cc:648] HybridClock initialized: now 1786177245544117 us; error 48 us; skew 500 ppm
I20260808 08:20:45.546339 26660 webserver.cc:460] Webserver started at http://127.12.13.2:34903/ using document root <none> and password file <none>
I20260808 08:20:45.547082 26660 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:45.547150 26660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:45.547394 26660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:45.549517 26660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "ac4ff928f115402ba0b362ccd491dc6e"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "d7f00897c129f213b804a8bda7dda4ad"
server_key_iv: "7f8a5d1f5a655bacd15952247b1668b3"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.550109 26660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "ac4ff928f115402ba0b362ccd491dc6e"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "d7f00897c129f213b804a8bda7dda4ad"
server_key_iv: "7f8a5d1f5a655bacd15952247b1668b3"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.554133 26660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:20:45.556766 26681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.558097 26660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:45.558220 26660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "ac4ff928f115402ba0b362ccd491dc6e"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "d7f00897c129f213b804a8bda7dda4ad"
server_key_iv: "7f8a5d1f5a655bacd15952247b1668b3"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.558331 26660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:45.577338 26660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:45.578158 26660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:45.578361 26660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:45.578994 26660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:45.579999 26660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:45.580046 26660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.580119 26660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:45.580161 26660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.590206 26660 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:33195
I20260808 08:20:45.590225 26845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:33195 every 8 connection(s)
I20260808 08:20:45.591187 26660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:45.593017 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 26660
I20260808 08:20:45.593163 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:45.593484 12340 external_mini_cluster.cc:1428] Setting key fdda22bdeb03d839922e82978df78e87
I20260808 08:20:45.595758 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40103
--builtin_ntp_servers=127.12.13.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 08:20:45.602377 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.593196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:42191 (local address 127.12.13.62:40103)
0808 08:20:45.593365 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:45.593370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:45.594777 (+ 1407us) server_negotiation.cc:407] Connection header received
0808 08:20:45.596491 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:45.596498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:45.596564 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:45.596660 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:45.597676 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.598288 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.600095 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.600306 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.600398 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.601809 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:45.601812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:45.601814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:45.601827 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:45.601848 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:45.601858 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:45.601982 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:45.602188 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 08:20:45.602242 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:20:45.603817 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.593730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:40103 (local address 127.12.13.2:42191)
0808 08:20:45.594593 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:45.594675 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:45.596250 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:45.596827 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:45.596836 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:45.596924 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:45.597485 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:45.597509 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.598456 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.598462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:45.599950 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:45.599965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.600470 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.600477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:45.600760 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.601463 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:45.601468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:45.601533 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:45.601634 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:45.602013 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:45.602025 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:45.602035 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 08:20:45.602191 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 08:20:45.602565 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":185,"threads_started":1}
I20260808 08:20:45.605176 26846 heartbeater.cc:347] Connected to a master server at 127.12.13.62:40103
I20260808 08:20:45.605444 26846 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:45.606129 26846 heartbeater.cc:511] Master 127.12.13.62:40103 requested a full tablet report, sending...
I20260808 08:20:45.607420 26389 ts_manager.cc:194] Registered new tserver with Master: ac4ff928f115402ba0b362ccd491dc6e (127.12.13.2:33195)
I20260808 08:20:45.608361 26389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.2:42191
W20260808 08:20:45.718765 26854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:45.719049 26854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:45.719089 26854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:45.723146 26854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:45.723202 26854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:45.723254 26854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:45.727977 26854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:45.729074 26854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:45.730571 26854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:45.740881 26860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:45.740866 26861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:45.741056 26854 server_base.cc:1034] running on GCE node
W20260808 08:20:45.740895 26865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:45.741843 26854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:45.742597 26854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:45.743810 26854 hybrid_clock.cc:648] HybridClock initialized: now 1786177245743757 us; error 46 us; skew 500 ppm
I20260808 08:20:45.745754 26854 webserver.cc:460] Webserver started at http://127.12.13.3:35129/ using document root <none> and password file <none>
I20260808 08:20:45.746493 26854 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:45.746608 26854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:45.746827 26854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:45.749060 26854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "1bfddcdb7f7c4938a6969ffe6ad61a0b"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "6e62b1101fe3250136745b4de9bd7c7e"
server_key_iv: "082bc92590b2326360e35e0b65f1d6f4"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.749584 26854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "1bfddcdb7f7c4938a6969ffe6ad61a0b"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "6e62b1101fe3250136745b4de9bd7c7e"
server_key_iv: "082bc92590b2326360e35e0b65f1d6f4"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.753600 26854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 08:20:45.756325 26871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.757588 26854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:45.757721 26854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "1bfddcdb7f7c4938a6969ffe6ad61a0b"
format_stamp: "Formatted at 2026-08-08 08:20:45 on dist-test-slave-6ntk"
server_key: "6e62b1101fe3250136745b4de9bd7c7e"
server_key_iv: "082bc92590b2326360e35e0b65f1d6f4"
server_key_version: "encryptionkey@0"
I20260808 08:20:45.757843 26854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:45.784417 26854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:45.785379 26854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:45.785617 26854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:45.786842 26854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:45.787858 26854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:45.787909 26854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.787951 26854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:45.787966 26854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:45.797657 26854 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:46343
I20260808 08:20:45.797698 27056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:46343 every 8 connection(s)
I20260808 08:20:45.798861 26854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:45.804560 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 26854
I20260808 08:20:45.804672 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:45.804975 12340 external_mini_cluster.cc:1428] Setting key 44489b3a35c90f2b1c5e7167c3975654
I20260808 08:20:45.808542 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.800783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:37109 (local address 127.12.13.62:40103)
0808 08:20:45.800939 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:45.800944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:45.802155 (+ 1211us) server_negotiation.cc:407] Connection header received
0808 08:20:45.803501 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:45.803504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:45.803551 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:45.803626 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:45.804598 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.805310 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.806703 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.806926 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.807037 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.808120 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:45.808123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:45.808125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:45.808137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:45.808156 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:45.808163 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:45.808222 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:45.808377 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 08:20:45.808429 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:20:45.809398 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:45.801253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:40103 (local address 127.12.13.3:37109)
0808 08:20:45.802000 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:45.802043 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:45.803307 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:45.803793 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:45.803800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:45.803912 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:45.804449 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:45.804464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.805447 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.805451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:45.806588 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:45.806600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:45.807087 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:45.807092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:45.807296 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:45.807875 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:45.807876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:45.807917 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:45.807981 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:45.808239 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:45.808246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:45.808250 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 08:20:45.808352 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 08:20:45.808580 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260808 08:20:45.810578 27058 heartbeater.cc:347] Connected to a master server at 127.12.13.62:40103
I20260808 08:20:45.810810 27058 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:45.811364 27058 heartbeater.cc:511] Master 127.12.13.62:40103 requested a full tablet report, sending...
I20260808 08:20:45.812389 26389 ts_manager.cc:194] Registered new tserver with Master: 1bfddcdb7f7c4938a6969ffe6ad61a0b (127.12.13.3:46343)
I20260808 08:20:45.812971 26389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.3:37109
I20260808 08:20:45.819131 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 08:20:45.821908 26445 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:45.821029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.12.13.62:40103)
0808 08:20:45.821145 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:45.821148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:45.821325 (+ 177us) server_negotiation.cc:407] Connection header received
0808 08:20:45.821440 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:45.821442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:45.821481 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:45.821549 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:45.821765 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34184: BlockingRecv error: recv got EOF from 127.0.0.1:34184 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 08:20:45.822135 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 26475
I20260808 08:20:45.844681 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 26660
I20260808 08:20:45.856297 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 26854
I20260808 08:20:45.871348 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 26345
2026-08-08T08:20:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (975 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T08:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:45Z Disabled control of system clock
I20260808 08:20:45.911542 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43585
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 08:20:46.030148 27076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:46.030437 27076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:46.030476 27076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:46.034479 27076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:46.034538 27076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:46.034552 27076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:46.034569 27076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:46.034585 27076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:46.039481 27076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:43585
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:46.040668 27076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:46.042161 27076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:46.051265 27085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:46.051206 27092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:46.051206 27087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:46.051486 27076 server_base.cc:1034] running on GCE node
I20260808 08:20:46.052177 27076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:46.053385 27076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:46.054594 27076 hybrid_clock.cc:648] HybridClock initialized: now 1786177246054579 us; error 27 us; skew 500 ppm
I20260808 08:20:46.056506 27076 webserver.cc:460] Webserver started at http://127.12.13.62:41245/ using document root <none> and password file <none>
I20260808 08:20:46.057142 27076 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:46.057240 27076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:46.057494 27076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:46.059737 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "7a451d9935524ecca5c407c8e08c9dee"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "6196be033778c603f17837290535e1c0"
server_key_iv: "3819e4704e5990395eaa10b18ee5ec25"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.060361 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "7a451d9935524ecca5c407c8e08c9dee"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "6196be033778c603f17837290535e1c0"
server_key_iv: "3819e4704e5990395eaa10b18ee5ec25"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.064467 27076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:46.067201 27101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.068531 27076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:20:46.068702 27076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "7a451d9935524ecca5c407c8e08c9dee"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "6196be033778c603f17837290535e1c0"
server_key_iv: "3819e4704e5990395eaa10b18ee5ec25"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.068840 27076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:46.073755 27076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:46.074438 27076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:46.074652 27076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:46.082170 27177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:43585 every 8 connection(s)
I20260808 08:20:46.082166 27076 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:43585
I20260808 08:20:46.083283 27076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:46.086948 27180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:46.088280 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 27076
I20260808 08:20:46.088416 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:46.088732 12340 external_mini_cluster.cc:1428] Setting key 4bbc94291d52ec29db521d032f1fcbea
I20260808 08:20:46.093436 27180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee: Bootstrap starting.
I20260808 08:20:46.094698 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.090359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.12.13.62:43585)
0808 08:20:46.091053 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:46.091069 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:46.091106 (+ 37us) server_negotiation.cc:407] Connection header received
0808 08:20:46.091766 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:46.091784 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:46.092139 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:46.092497 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:46.093337 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:46.093342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:46.093354 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:46.093366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:46.093425 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:46.093441 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:46.093604 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:46.093679 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 08:20:46.093980 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260808 08:20:46.096166 27180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:46.097000 27180 log.cc:824] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:46.099022 27180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee: No bootstrap required, opened a new log
I20260808 08:20:46.103607 27180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a451d9935524ecca5c407c8e08c9dee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43585 } }
I20260808 08:20:46.103863 27180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:46.103943 27180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a451d9935524ecca5c407c8e08c9dee, State: Initialized, Role: FOLLOWER
I20260808 08:20:46.104437 27180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a451d9935524ecca5c407c8e08c9dee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43585 } }
I20260808 08:20:46.104606 27180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:46.104674 27180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:46.104810 27180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:46.105983 27180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a451d9935524ecca5c407c8e08c9dee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43585 } }
I20260808 08:20:46.106355 27180 leader_election.cc:302] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a451d9935524ecca5c407c8e08c9dee; no voters:
I20260808 08:20:46.106732 27180 leader_election.cc:288] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:46.106856 27188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:46.107172 27188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [term 1 LEADER]: Becoming Leader. State: Replica: 7a451d9935524ecca5c407c8e08c9dee, State: Running, Role: LEADER
I20260808 08:20:46.107537 27188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a451d9935524ecca5c407c8e08c9dee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43585 } }
I20260808 08:20:46.108158 27180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:46.109429 27189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a451d9935524ecca5c407c8e08c9dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a451d9935524ecca5c407c8e08c9dee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43585 } } }
I20260808 08:20:46.109436 27190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a451d9935524ecca5c407c8e08c9dee. Latest consensus state: current_term: 1 leader_uuid: "7a451d9935524ecca5c407c8e08c9dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a451d9935524ecca5c407c8e08c9dee" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43585 } } }
I20260808 08:20:46.109568 27189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:46.109572 27190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:46.110061 27203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:46.113174 27203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:46.119287 27203 catalog_manager.cc:1347] Generated new cluster ID: d26fbc114326485cbeb7e91ae36a6a3c
I20260808 08:20:46.119370 27203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:46.131925 27203 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:46.132349 27203 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:46.140112 27203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a451d9935524ecca5c407c8e08c9dee: Generated new TSK 0
I20260808 08:20:46.140974 27203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:20:46.146499 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43585
--builtin_ntp_servers=127.12.13.20:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 08:20:46.272643 27222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:46.272938 27222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:46.273020 27222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:46.277251 27222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:46.277326 27222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:46.277390 27222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:46.282318 27222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:46.283599 27222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:46.285179 27222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:46.296181 27227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:46.296228 27229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:46.296422 27232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:46.296267 27222 server_base.cc:1034] running on GCE node
I20260808 08:20:46.296885 27222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:46.297504 27222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:46.298661 27222 hybrid_clock.cc:648] HybridClock initialized: now 1786177246298624 us; error 49 us; skew 500 ppm
I20260808 08:20:46.300624 27222 webserver.cc:460] Webserver started at http://127.12.13.1:33101/ using document root <none> and password file <none>
I20260808 08:20:46.301232 27222 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:46.301295 27222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:46.301570 27222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:46.303706 27222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "7e0a586cf7294c2799347c103d6df3cc"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "29f6c1bf7f35e0e983f4cf6c51c5cbbc"
server_key_iv: "dc6879aadc66af6e6f1b45857144ad81"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.304270 27222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "7e0a586cf7294c2799347c103d6df3cc"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "29f6c1bf7f35e0e983f4cf6c51c5cbbc"
server_key_iv: "dc6879aadc66af6e6f1b45857144ad81"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.308117 27222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 08:20:46.310985 27242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.312451 27222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:20:46.312585 27222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "7e0a586cf7294c2799347c103d6df3cc"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "29f6c1bf7f35e0e983f4cf6c51c5cbbc"
server_key_iv: "dc6879aadc66af6e6f1b45857144ad81"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.312711 27222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:46.317452 27222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:46.318140 27222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:46.318328 27222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:46.318887 27222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:46.319818 27222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:46.319865 27222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.319934 27222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:46.319965 27222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.328960 27222 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:44639
I20260808 08:20:46.329056 27416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:44639 every 8 connection(s)
I20260808 08:20:46.329981 27222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:46.335059 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 27222
I20260808 08:20:46.335201 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:46.335510 12340 external_mini_cluster.cc:1428] Setting key 03dceb95551fcac3a9dee5467befe196
I20260808 08:20:46.336433 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.331870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56159 (local address 127.12.13.62:43585)
0808 08:20:46.332027 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:46.332032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:46.333168 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 08:20:46.334528 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:46.334530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:46.334570 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:46.334646 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:46.335900 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:46.335903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:46.335906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:46.335927 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:46.335949 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:46.335958 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:46.336040 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:46.336262 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 08:20:46.336311 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:20:46.337453 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.332339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43585 (local address 127.12.13.1:56159)
0808 08:20:46.333019 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:46.333059 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:46.334337 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:46.334804 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:46.334811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:46.334907 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:46.335571 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:46.335573 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:46.335647 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:46.335732 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:46.336051 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:46.336062 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:46.336072 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 08:20:46.336273 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 08:20:46.336556 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260808 08:20:46.337702 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43585
--builtin_ntp_servers=127.12.13.20:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 08:20:46.338681 27418 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43585
I20260808 08:20:46.338946 27418 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:46.339443 27418 heartbeater.cc:511] Master 127.12.13.62:43585 requested a full tablet report, sending...
I20260808 08:20:46.341293 27128 ts_manager.cc:194] Registered new tserver with Master: 7e0a586cf7294c2799347c103d6df3cc (127.12.13.1:44639)
W20260808 08:20:46.455816 27424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:46.456135 27424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:46.456210 27424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:46.460201 27424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:46.460284 27424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:46.460381 27424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:46.465123 27424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:46.466297 27424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:46.467749 27424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:46.478214 27438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:46.478262 27424 server_base.cc:1034] running on GCE node
W20260808 08:20:46.478206 27436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:46.478206 27435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:46.478889 27424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:46.479534 27424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:46.480700 27424 hybrid_clock.cc:648] HybridClock initialized: now 1786177246480685 us; error 26 us; skew 500 ppm
I20260808 08:20:46.482661 27424 webserver.cc:460] Webserver started at http://127.12.13.2:43575/ using document root <none> and password file <none>
I20260808 08:20:46.483276 27424 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:46.483374 27424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:46.483623 27424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:46.485737 27424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "071571c8cb2e4fcb88bae45d49119628"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "c1eb0a50a834c457bbaa007f4b9b5fd7"
server_key_iv: "3d8804a1ada949589cbb9dff4e08b44a"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.486452 27424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "071571c8cb2e4fcb88bae45d49119628"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "c1eb0a50a834c457bbaa007f4b9b5fd7"
server_key_iv: "3d8804a1ada949589cbb9dff4e08b44a"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.490406 27424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:46.493009 27447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.494313 27424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:46.494455 27424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "071571c8cb2e4fcb88bae45d49119628"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "c1eb0a50a834c457bbaa007f4b9b5fd7"
server_key_iv: "3d8804a1ada949589cbb9dff4e08b44a"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.494565 27424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:46.499492 27424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:46.500092 27424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:46.500272 27424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:46.500839 27424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:46.501775 27424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:46.501822 27424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.501914 27424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:46.501951 27424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.511186 27424 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:38483
I20260808 08:20:46.511210 27619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:38483 every 8 connection(s)
I20260808 08:20:46.512192 27424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:46.516021 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 27424
I20260808 08:20:46.516165 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:46.516485 12340 external_mini_cluster.cc:1428] Setting key ebc1207a821eee7d91802a5561b175fd
I20260808 08:20:46.518924 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43585
--builtin_ntp_servers=127.12.13.20:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 08:20:46.519230 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.514214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:42161 (local address 127.12.13.62:43585)
0808 08:20:46.514407 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:46.514412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:46.515546 (+ 1134us) server_negotiation.cc:407] Connection header received
0808 08:20:46.517271 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:46.517274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:46.517320 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:46.517407 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:46.518705 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:46.518708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:46.518710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:46.518721 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:46.518740 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:46.518749 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:46.518822 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:46.519024 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 08:20:46.519076 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 08:20:46.520385 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.514684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43585 (local address 127.12.13.2:42161)
0808 08:20:46.515379 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:46.515427 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:46.517022 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:46.517614 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:46.517623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:46.517737 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:46.518397 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:46.518399 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:46.518455 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:46.518544 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:46.518841 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:46.518851 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:46.518857 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:20:46.519050 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 08:20:46.519311 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260808 08:20:46.521643 27620 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43585
I20260808 08:20:46.521934 27620 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:46.522388 27620 heartbeater.cc:511] Master 127.12.13.62:43585 requested a full tablet report, sending...
I20260808 08:20:46.523522 27128 ts_manager.cc:194] Registered new tserver with Master: 071571c8cb2e4fcb88bae45d49119628 (127.12.13.2:38483)
W20260808 08:20:46.639959 27624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:46.640412 27624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:46.640509 27624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:46.644724 27624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:46.644821 27624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:46.644917 27624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:46.649997 27624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:46.651223 27624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:46.652776 27624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:46.663328 27635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:46.663404 27638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:46.663600 27624 server_base.cc:1034] running on GCE node
W20260808 08:20:46.663404 27636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:46.664077 27624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:46.664839 27624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:46.666034 27624 hybrid_clock.cc:648] HybridClock initialized: now 1786177246666013 us; error 28 us; skew 500 ppm
I20260808 08:20:46.678704 27624 webserver.cc:460] Webserver started at http://127.12.13.3:36977/ using document root <none> and password file <none>
I20260808 08:20:46.679571 27624 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:46.679708 27624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:46.680065 27624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:46.682617 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "770802b24e3d46e1aa24948803581fa2"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "922909c9cc51780776966ec1dd094829"
server_key_iv: "f8fce8367d75458205588bf5dbafcffd"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.683271 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "770802b24e3d46e1aa24948803581fa2"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "922909c9cc51780776966ec1dd094829"
server_key_iv: "f8fce8367d75458205588bf5dbafcffd"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.687631 27624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:20:46.690642 27648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.692013 27624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:20:46.692176 27624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "770802b24e3d46e1aa24948803581fa2"
format_stamp: "Formatted at 2026-08-08 08:20:46 on dist-test-slave-6ntk"
server_key: "922909c9cc51780776966ec1dd094829"
server_key_iv: "f8fce8367d75458205588bf5dbafcffd"
server_key_version: "encryptionkey@0"
I20260808 08:20:46.692324 27624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:46.697819 27624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:46.698580 27624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:46.698807 27624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:46.699448 27624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:46.700492 27624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:46.700574 27624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.700649 27624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:46.700706 27624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:46.711370 27624 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:41181
I20260808 08:20:46.711401 27824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:41181 every 8 connection(s)
I20260808 08:20:46.712553 27624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:46.717856 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 27624
I20260808 08:20:46.718037 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:46.718438 12340 external_mini_cluster.cc:1428] Setting key b80323e3e67b522d5cbc44ebf7236203
I20260808 08:20:46.719563 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.714547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:43383 (local address 127.12.13.62:43585)
0808 08:20:46.714701 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:46.714705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:46.715898 (+ 1193us) server_negotiation.cc:407] Connection header received
0808 08:20:46.717454 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:46.717457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:46.717497 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:46.717564 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:46.719004 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:46.719008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:46.719011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:20:46.719027 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:46.719054 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:20:46.719071 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:46.719179 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:46.719358 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 08:20:46.719412 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:20:46.720602 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:46.715008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43585 (local address 127.12.13.3:43383)
0808 08:20:46.715733 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:46.715784 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:46.717206 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:46.717760 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:46.717772 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:46.717930 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:46.718655 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:20:46.718658 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:46.718717 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:20:46.718813 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:46.719175 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:46.719186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:46.719196 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 08:20:46.719409 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 08:20:46.719681 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260808 08:20:46.721696 27825 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43585
I20260808 08:20:46.721974 27825 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:46.722424 27825 heartbeater.cc:511] Master 127.12.13.62:43585 requested a full tablet report, sending...
I20260808 08:20:46.723477 27128 ts_manager.cc:194] Registered new tserver with Master: 770802b24e3d46e1aa24948803581fa2 (127.12.13.3:41181)
I20260808 08:20:46.732758 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 08:20:46.736122 27185 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:20:46.734895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.12.13.62:43585)
0808 08:20:46.735092 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:46.735098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:46.735266 (+ 168us) server_negotiation.cc:407] Connection header received
0808 08:20:46.735425 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:46.735429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:46.735472 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:46.735563 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:46.735971 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36490: BlockingRecv error: recv got EOF from 127.0.0.1:36490 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:20:46.736274 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 27222
I20260808 08:20:46.761200 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 27424
I20260808 08:20:46.777321 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 27624
I20260808 08:20:46.802242 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 27076
2026-08-08T08:20:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (928 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:46 dist-test-slave-6ntk krb5kdc[27843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:46 dist-test-slave-6ntk krb5kdc[27843](info): set up 2 sockets
Aug 08 08:20:46 dist-test-slave-6ntk krb5kdc[27843](info): commencing operation
krb5kdc: starting...
W20260808 08:20:48.059121 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:20:48.195715 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:44813
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:44813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:48.313385 27868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:48.313726 27868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:48.313802 27868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:48.317940 27868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:20:48.318049 27868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:48.318084 27868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:48.318123 27868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:20:48.318156 27868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:20:48.323623 27868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:44813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:48.324832 27868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:48.326186 27868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:48.335760 27879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:48.335733 27882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:48.335764 27880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:48.335969 27868 server_base.cc:1034] running on GCE node
I20260808 08:20:48.336688 27868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:48.337975 27868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:48.339231 27868 hybrid_clock.cc:648] HybridClock initialized: now 1786177248339200 us; error 49 us; skew 500 ppm
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:48.342733 27868 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:20:48.343866 27868 webserver.cc:460] Webserver started at http://127.12.13.62:33811/ using document root <none> and password file <none>
I20260808 08:20:48.344555 27868 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:48.344678 27868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:48.344971 27868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:48.347509 27868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "b8c5832353c04b94baebc213010751db"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "50e3619a404bf8042f476f9c44e361c3"
server_key_iv: "3d4079b9957b8fb19b3bc8e22254f347"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.348203 27868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "b8c5832353c04b94baebc213010751db"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "50e3619a404bf8042f476f9c44e361c3"
server_key_iv: "3d4079b9957b8fb19b3bc8e22254f347"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.352687 27868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:20:48.355602 27892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.357057 27868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:48.357232 27868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "b8c5832353c04b94baebc213010751db"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "50e3619a404bf8042f476f9c44e361c3"
server_key_iv: "3d4079b9957b8fb19b3bc8e22254f347"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.357393 27868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:48.402123 27868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:48.406049 27868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:48.406298 27868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:48.414199 27868 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:44813
I20260808 08:20:48.414503 27975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:44813 every 8 connection(s)
I20260808 08:20:48.415400 27868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:20:48.418473 27976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:48.424074 27976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: Bootstrap starting.
I20260808 08:20:48.424515 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 27868
I20260808 08:20:48.424643 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:20:48.424948 12340 external_mini_cluster.cc:1428] Setting key 7ac94bb06a61d22e056d45b66ec94be9
I20260808 08:20:48.427000 27976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:48.427774 27976 log.cc:824] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:48.429677 27976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: No bootstrap required, opened a new log
I20260808 08:20:48.433211 27976 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5832353c04b94baebc213010751db" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44813 } }
I20260808 08:20:48.433468 27976 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:48.433552 27976 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c5832353c04b94baebc213010751db, State: Initialized, Role: FOLLOWER
I20260808 08:20:48.434083 27976 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5832353c04b94baebc213010751db" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44813 } }
I20260808 08:20:48.434237 27976 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:20:48.434314 27976 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:20:48.434423 27976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 0 FOLLOWER]: Advancing to term 1
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:48.435438 27976 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5832353c04b94baebc213010751db" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44813 } }
I20260808 08:20:48.435786 27976 leader_election.cc:302] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c5832353c04b94baebc213010751db; no voters:
I20260808 08:20:48.436127 27976 leader_election.cc:288] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:20:48.436594 27987 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5832353c04b94baebc213010751db" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44813 } } }
I20260808 08:20:48.436636 27986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:48.436733 27987 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:20:48.437418 27976 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:20:48.438056 27986 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [term 1 LEADER]: Becoming Leader. State: Replica: b8c5832353c04b94baebc213010751db, State: Running, Role: LEADER
I20260808 08:20:48.438467 27986 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5832353c04b94baebc213010751db" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44813 } }
I20260808 08:20:48.440290 27987 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c5832353c04b94baebc213010751db. Latest consensus state: current_term: 1 leader_uuid: "b8c5832353c04b94baebc213010751db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5832353c04b94baebc213010751db" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 44813 } } }
I20260808 08:20:48.440798 27987 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db [sys.catalog]: This master's current role is: LEADER
I20260808 08:20:48.442252 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:48.426460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.12.13.62:44813)
0808 08:20:48.426930 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:48.426947 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:48.426986 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:20:48.427785 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:48.427803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:48.428222 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:48.428584 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:48.429538 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.430793 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.431739 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.432264 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.432461 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:48.435527 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:48.435536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:48.435558 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:48.435597 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:48.437738 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:48.438402 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:48.438411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:48.438420 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:48.438582 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:48.438946 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:48.438952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:48.438954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:48.439553 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:48.439779 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:48.440809 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0808 08:20:48.441136 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
W20260808 08:20:48.443434 28008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:20:48.443516 28008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:20:48.443595 28009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:20:48.444399 28009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:20:48.450519 28009 catalog_manager.cc:1347] Generated new cluster ID: 48febdbefbe147d1829c5ddbdd5f91b3
I20260808 08:20:48.450588 28009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:20:48.460615 28009 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:20:48.462020 28009 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:20:48.468971 28009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: Generated new TSK 0
I20260808 08:20:48.469755 28009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:20:48.505622 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44813
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:48.631580 28018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:48.632035 28018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:48.632125 28018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:48.636300 28018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:48.636404 28018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:48.636502 28018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:20:48.642499 28018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:48.643751 28018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:48.644946 28018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:48.655489 28031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:48.655565 28026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:48.655597 28018 server_base.cc:1034] running on GCE node
W20260808 08:20:48.655519 28024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:48.656275 28018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:48.656942 28018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:48.658142 28018 hybrid_clock.cc:648] HybridClock initialized: now 1786177248658129 us; error 32 us; skew 500 ppm
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:48.660987 28018 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:20:48.662016 28018 webserver.cc:460] Webserver started at http://127.12.13.1:35079/ using document root <none> and password file <none>
I20260808 08:20:48.662644 28018 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:48.662736 28018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:48.662949 28018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:48.665033 28018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "b139c45c157a4ee48b4028840efd5a9b"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "e437f826f4aea1ae93243f879b4cbd60"
server_key_iv: "3093d6941ecaa97b4ffe407daa1fbb14"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.665586 28018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "b139c45c157a4ee48b4028840efd5a9b"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "e437f826f4aea1ae93243f879b4cbd60"
server_key_iv: "3093d6941ecaa97b4ffe407daa1fbb14"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.669549 28018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 08:20:48.672185 28043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.673499 28018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:20:48.673629 28018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "b139c45c157a4ee48b4028840efd5a9b"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "e437f826f4aea1ae93243f879b4cbd60"
server_key_iv: "3093d6941ecaa97b4ffe407daa1fbb14"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.673744 28018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:48.690343 28018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:48.693465 28018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:48.693683 28018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:48.694402 28018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:48.695279 28018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:48.695354 28018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.695472 28018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:48.695508 28018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.704949 28018 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:46261
I20260808 08:20:48.704970 28237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:46261 every 8 connection(s)
I20260808 08:20:48.705994 28018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:20:48.714442 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 28018
I20260808 08:20:48.714609 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:20:48.714933 12340 external_mini_cluster.cc:1428] Setting key ce1dd20cde848b84b90e15adb166974a
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:48.722191 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:48.708051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59075 (local address 127.12.13.62:44813)
0808 08:20:48.708197 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:48.708201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:48.709059 (+ 858us) server_negotiation.cc:407] Connection header received
0808 08:20:48.710666 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:48.710670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:48.710737 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:48.710851 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:48.712168 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.712726 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.714005 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.714267 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.714403 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:48.717499 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:48.717506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:48.717512 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:48.717550 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:48.719453 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:48.720297 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:48.720300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:48.720301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:48.720393 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:48.720840 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:48.720843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:48.720845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:48.721064 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:48.721147 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:48.721936 (+ 789us) server_negotiation.cc:299] Negotiation successful
0808 08:20:48.722059 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:20:48.723562 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:48.708396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44813 (local address 127.12.13.1:59075)
0808 08:20:48.708893 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:48.708941 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:48.710422 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:48.711062 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:48.711070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:48.711472 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:48.712002 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:48.712017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.712874 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.712880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:48.713788 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:48.713798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.714456 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.714460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:48.714704 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:48.715460 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:48.715478 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:48.717298 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:48.719602 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:48.719613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:48.719636 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:48.720099 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:48.720505 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:48.720511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:48.720515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:48.720729 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:48.721183 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:48.721194 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:48.721587 (+ 393us) client_negotiation.cc:769] Sending connection context
0808 08:20:48.721825 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 08:20:48.722273 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":124,"threads_started":1}
I20260808 08:20:48.725100 28238 heartbeater.cc:347] Connected to a master server at 127.12.13.62:44813
I20260808 08:20:48.725418 28238 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:48.726248 28238 heartbeater.cc:511] Master 127.12.13.62:44813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:20:48.728127 27922 ts_manager.cc:194] Registered new tserver with Master: b139c45c157a4ee48b4028840efd5a9b (127.12.13.1:46261)
I20260808 08:20:48.729472 27922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:59075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:20:48.749305 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44813
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:48.876164 28245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:48.876534 28245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:48.876585 28245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:48.880740 28245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:48.880806 28245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:48.880865 28245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:20:48.886442 28245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:48.887601 28245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:48.888640 28245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:48.899299 28256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:48.899312 28252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:48.899555 28253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:48.899739 28245 server_base.cc:1034] running on GCE node
I20260808 08:20:48.900198 28245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:48.900911 28245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:48.902132 28245 hybrid_clock.cc:648] HybridClock initialized: now 1786177248902109 us; error 31 us; skew 500 ppm
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:48.905019 28245 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:20:48.906051 28245 webserver.cc:460] Webserver started at http://127.12.13.2:43171/ using document root <none> and password file <none>
I20260808 08:20:48.906622 28245 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:48.906680 28245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:48.906876 28245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:48.909180 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "49578faacad8494098af1dfdf9e8837f"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "a5080832f0cec9b3460386d66b871faf"
server_key_iv: "622467a0c26e8fa498f4e404d9045b1e"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.909761 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "49578faacad8494098af1dfdf9e8837f"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "a5080832f0cec9b3460386d66b871faf"
server_key_iv: "622467a0c26e8fa498f4e404d9045b1e"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.913676 28245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 08:20:48.916491 28264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.917914 28245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:20:48.918054 28245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "49578faacad8494098af1dfdf9e8837f"
format_stamp: "Formatted at 2026-08-08 08:20:48 on dist-test-slave-6ntk"
server_key: "a5080832f0cec9b3460386d66b871faf"
server_key_iv: "622467a0c26e8fa498f4e404d9045b1e"
server_key_version: "encryptionkey@0"
I20260808 08:20:48.918177 28245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:48.934340 28245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:48.937410 28245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:48.937624 28245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:48.938483 28245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:48.939379 28245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:48.939450 28245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.939517 28245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:48.939550 28245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:48.949249 28245 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:41827
I20260808 08:20:48.949271 28431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:41827 every 8 connection(s)
I20260808 08:20:48.950269 28245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:20:48.958050 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 28245
I20260808 08:20:48.958194 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:20:48.958505 12340 external_mini_cluster.cc:1428] Setting key 8f222218dae4e3996c29acfc41ad3585
Aug 08 08:20:48 dist-test-slave-6ntk krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177248, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:48.965494 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:48.952262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:60079 (local address 127.12.13.62:44813)
0808 08:20:48.952406 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:48.952410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:48.953276 (+ 866us) server_negotiation.cc:407] Connection header received
0808 08:20:48.954816 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:48.954819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:48.954901 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:48.955006 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:48.956221 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.956772 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.958019 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.958329 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.958449 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:48.961311 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:48.961315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:48.961318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:48.961348 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:48.963244 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:48.963798 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:48.963804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:48.963807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:48.963907 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:48.964343 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:48.964345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:48.964346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:48.964533 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:48.964615 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:48.965269 (+ 654us) server_negotiation.cc:299] Negotiation successful
0808 08:20:48.965372 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 08:20:48.966864 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:48.952667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44813 (local address 127.12.13.2:60079)
0808 08:20:48.953126 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:48.953169 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:48.954592 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:48.955192 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:48.955199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:48.955575 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:48.956065 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:48.956079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.956941 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.956943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:48.957839 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:48.957849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:48.958494 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:48.958498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:48.958734 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:48.959394 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:48.959411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:48.961122 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:48.963395 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:48.963401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:48.963414 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:48.963689 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:48.964035 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:48.964041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:48.964046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:48.964242 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:48.964643 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:48.964652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:48.964998 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 08:20:48.965227 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 08:20:48.965651 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":134,"threads_started":1}
I20260808 08:20:48.968072 28432 heartbeater.cc:347] Connected to a master server at 127.12.13.62:44813
I20260808 08:20:48.968327 28432 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:48.968920 28432 heartbeater.cc:511] Master 127.12.13.62:44813 requested a full tablet report, sending...
I20260808 08:20:48.970134 27922 ts_manager.cc:194] Registered new tserver with Master: 49578faacad8494098af1dfdf9e8837f (127.12.13.2:41827)
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:20:48.970844 27922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:60079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:20:48.992924 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44813
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:20:49.114805 28440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:20:49.115211 28440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:20:49.115336 28440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:20:49.119670 28440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:20:49.119769 28440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:20:49.119868 28440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:20:49.125634 28440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:20:49.126960 28440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:20:49.128248 28440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:20:49.139186 28454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:49.139191 28448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:20:49.139176 28450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:20:49.139525 28440 server_base.cc:1034] running on GCE node
I20260808 08:20:49.140187 28440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:20:49.140934 28440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:20:49.142115 28440 hybrid_clock.cc:648] HybridClock initialized: now 1786177249142076 us; error 51 us; skew 500 ppm
Aug 08 08:20:49 dist-test-slave-6ntk krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177249, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:20:49.145212 28440 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:20:49.146389 28440 webserver.cc:460] Webserver started at http://127.12.13.3:40161/ using document root <none> and password file <none>
I20260808 08:20:49.147053 28440 fs_manager.cc:359] Metadata directory not provided
I20260808 08:20:49.147157 28440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:20:49.147420 28440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:20:49.149736 28440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "0f95737ff9164601ada9728ec4241a4f"
format_stamp: "Formatted at 2026-08-08 08:20:49 on dist-test-slave-6ntk"
server_key: "5fb37d80e61b49325be95e49110b8574"
server_key_iv: "1b009c14fd1a29b77c26e684d8cb7284"
server_key_version: "encryptionkey@0"
I20260808 08:20:49.150466 28440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "0f95737ff9164601ada9728ec4241a4f"
format_stamp: "Formatted at 2026-08-08 08:20:49 on dist-test-slave-6ntk"
server_key: "5fb37d80e61b49325be95e49110b8574"
server_key_iv: "1b009c14fd1a29b77c26e684d8cb7284"
server_key_version: "encryptionkey@0"
I20260808 08:20:49.154708 28440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:20:49.157390 28470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:49.158725 28440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:20:49.158908 28440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "0f95737ff9164601ada9728ec4241a4f"
format_stamp: "Formatted at 2026-08-08 08:20:49 on dist-test-slave-6ntk"
server_key: "5fb37d80e61b49325be95e49110b8574"
server_key_iv: "1b009c14fd1a29b77c26e684d8cb7284"
server_key_version: "encryptionkey@0"
I20260808 08:20:49.159047 28440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:20:49.187057 28440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:20:49.190294 28440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:20:49.190477 28440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:20:49.191170 28440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:20:49.192071 28440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:20:49.192122 28440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:49.192166 28440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:20:49.192181 28440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:20:49.201840 28440 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:41011
I20260808 08:20:49.201927 28645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:41011 every 8 connection(s)
I20260808 08:20:49.203037 28440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:20:49.211338 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 28440
I20260808 08:20:49.211549 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:20:49.211956 12340 external_mini_cluster.cc:1428] Setting key 759957aacc31631871c374633b21af5e
Aug 08 08:20:49 dist-test-slave-6ntk krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177249, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:20:49.219982 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.205213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:38541 (local address 127.12.13.62:44813)
0808 08:20:49.205398 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.205403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.206473 (+ 1070us) server_negotiation.cc:407] Connection header received
0808 08:20:49.208074 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.208077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.208134 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.208228 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:49.209599 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.210312 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.211720 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.212136 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.212284 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.215749 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:49.215753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:49.215755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:49.215784 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:49.217507 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:49.218263 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:49.218266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:49.218267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:49.218374 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:49.218778 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:49.218781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:49.218782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:49.219012 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:49.219097 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.219719 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.219833 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:20:49.220969 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.205717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:44813 (local address 127.12.13.3:38541)
0808 08:20:49.206281 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.206329 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:49.207802 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:49.208447 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:49.208455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:49.208866 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:20:49.209418 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.209433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.210469 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.210472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.211550 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.211565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.212412 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.212417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.212662 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.213652 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:20:49.213671 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:20:49.215557 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:20:49.217678 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:49.217685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:49.217702 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:49.218124 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:49.218478 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:20:49.218481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:20:49.218482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:20:49.218652 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:20:49.219110 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:20:49.219116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:20:49.219475 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 08:20:49.219650 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 08:20:49.219956 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":184,"threads_started":1}
I20260808 08:20:49.222419 28646 heartbeater.cc:347] Connected to a master server at 127.12.13.62:44813
I20260808 08:20:49.222658 28646 heartbeater.cc:464] Registering TS with master...
I20260808 08:20:49.223179 28646 heartbeater.cc:511] Master 127.12.13.62:44813 requested a full tablet report, sending...
I20260808 08:20:49.224280 27922 ts_manager.cc:194] Registered new tserver with Master: 0f95737ff9164601ada9728ec4241a4f (127.12.13.3:41011)
I20260808 08:20:49.224947 27922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:38541
I20260808 08:20:49.226764 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:20:49.237329 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.228986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.12.13.62:44813)
0808 08:20:49.229119 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.229126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.229358 (+ 232us) server_negotiation.cc:407] Connection header received
0808 08:20:49.229470 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.229472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.229522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.229592 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:20:49.230651 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.231215 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.232145 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.232391 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.232494 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.234024 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:20:49.234027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:20:49.234029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:20:49.234055 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:20:49.235685 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:49.236143 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:49.236146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:49.236147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:49.236231 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:20:49.236532 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:20:49.236535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:20:49.236537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:20:49.236777 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:20:49.236917 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.237089 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.237196 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 08:20:49.240832 27922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:20:49.244171 27922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:20:49.260238 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.254020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.12.13.2:41827)
0808 08:20:49.255374 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.255389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.255408 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:20:49.255476 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.255480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.255687 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.255856 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:49.256131 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.256866 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.258565 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.259450 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.259574 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.259765 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.259942 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.260054 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":141,"threads_started":1}
I20260808 08:20:49.260234 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.253726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46261 (local address 127.0.0.1:54000)
0808 08:20:49.254591 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.254646 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:49.254793 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:49.255740 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:49.255742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:49.255760 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:49.255951 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.255959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.257359 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.257362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.258800 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.258807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.259852 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.259858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.259960 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.259990 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 08:20:49.260056 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 08:20:49.260108 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":303,"thread_start_us":130,"threads_started":1}
I20260808 08:20:49.260238 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.254049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:41011 (local address 127.0.0.1:41384)
0808 08:20:49.254641 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.254655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:49.254792 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:49.255582 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:49.255588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:49.255613 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:49.255778 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.255791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.257318 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.257322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.258643 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.258654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.259705 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.259709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.259804 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.259828 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:20:49.259920 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 08:20:49.260070 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":25,"thread_start_us":96,"threads_started":1}
I20260808 08:20:49.260775 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.253926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.12.13.1:46261)
0808 08:20:49.255084 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.255091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.255124 (+ 33us) server_negotiation.cc:407] Connection header received
0808 08:20:49.255204 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.255209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.255419 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.255575 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:49.256056 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.257236 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.258922 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.259476 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.259616 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.259785 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.260475 (+ 690us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.260605 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":86,"threads_started":1}
I20260808 08:20:49.260972 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.254610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.12.13.3:41011)
0808 08:20:49.255121 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.255131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.255149 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:20:49.255211 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.255215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.255419 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.255574 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:49.255925 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.257152 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.258771 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.259568 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.260030 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.260165 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.260215 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.260321 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":147,"threads_started":1}
I20260808 08:20:49.260864 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.253730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:41827 (local address 127.0.0.1:33104)
0808 08:20:49.255035 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.255051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:49.255135 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:49.255830 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:49.255833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:49.255854 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:49.255991 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.256001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.257002 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.257009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.258429 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.258438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.259603 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.259607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.259714 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.259754 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 08:20:49.259882 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 08:20:49.260724 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":688,"thread_start_us":106,"threads_started":1}
I20260808 08:20:49.264019 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 35541fe6efc447b89ca78687bffae77f (DEFAULT_TABLE table=test-table [id=ddaa14d5ce834881a559a85211172f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:49.264019 28551 tablet_service.cc:1467] Processing CreateTablet for tablet 35541fe6efc447b89ca78687bffae77f (DEFAULT_TABLE table=test-table [id=ddaa14d5ce834881a559a85211172f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:49.264034 28136 tablet_service.cc:1467] Processing CreateTablet for tablet 35541fe6efc447b89ca78687bffae77f (DEFAULT_TABLE table=test-table [id=ddaa14d5ce834881a559a85211172f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:20:49.265182 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35541fe6efc447b89ca78687bffae77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:49.265182 28136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35541fe6efc447b89ca78687bffae77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:49.265182 28551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35541fe6efc447b89ca78687bffae77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:20:49.271762 28671 tablet_bootstrap.cc:492] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: Bootstrap starting.
I20260808 08:20:49.273411 28674 tablet_bootstrap.cc:492] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Bootstrap starting.
I20260808 08:20:49.273911 28673 tablet_bootstrap.cc:492] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Bootstrap starting.
I20260808 08:20:49.274264 28671 tablet_bootstrap.cc:654] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:49.275122 28671 log.cc:824] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:49.276260 28674 tablet_bootstrap.cc:654] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:49.276441 28673 tablet_bootstrap.cc:654] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Neither blocks nor log segments found. Creating new log.
I20260808 08:20:49.277225 28674 log.cc:824] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:49.277218 28673 log.cc:824] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Log is configured to *not* fsync() on all Append() calls
I20260808 08:20:49.277437 28671 tablet_bootstrap.cc:492] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: No bootstrap required, opened a new log
I20260808 08:20:49.277678 28671 ts_tablet_manager.cc:1397] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:20:49.279433 28673 tablet_bootstrap.cc:492] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: No bootstrap required, opened a new log
I20260808 08:20:49.279433 28674 tablet_bootstrap.cc:492] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: No bootstrap required, opened a new log
I20260808 08:20:49.279628 28674 ts_tablet_manager.cc:1397] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:20:49.279628 28673 ts_tablet_manager.cc:1397] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:20:49.281396 28671 raft_consensus.cc:348] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.281651 28671 raft_consensus.cc:374] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:49.281704 28671 raft_consensus.cc:729] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b139c45c157a4ee48b4028840efd5a9b, State: Initialized, Role: FOLLOWER
I20260808 08:20:49.282235 28671 consensus_queue.cc:260] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.283332 28671 ts_tablet_manager.cc:1428] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260808 08:20:49.283589 28238 heartbeater.cc:503] Master 127.12.13.62:44813 was elected leader, sending a full tablet report...
I20260808 08:20:49.283568 28674 raft_consensus.cc:348] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.283568 28673 raft_consensus.cc:348] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.283847 28673 raft_consensus.cc:374] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:49.283847 28674 raft_consensus.cc:374] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:20:49.283918 28673 raft_consensus.cc:729] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49578faacad8494098af1dfdf9e8837f, State: Initialized, Role: FOLLOWER
I20260808 08:20:49.283922 28674 raft_consensus.cc:729] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f95737ff9164601ada9728ec4241a4f, State: Initialized, Role: FOLLOWER
I20260808 08:20:49.284379 28673 consensus_queue.cc:260] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.284451 28674 consensus_queue.cc:260] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.285377 28432 heartbeater.cc:503] Master 127.12.13.62:44813 was elected leader, sending a full tablet report...
I20260808 08:20:49.285401 28673 ts_tablet_manager.cc:1428] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:20:49.285401 28674 ts_tablet_manager.cc:1428] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:20:49.286100 28646 heartbeater.cc:503] Master 127.12.13.62:44813 was elected leader, sending a full tablet report...
W20260808 08:20:49.451525 28434 tablet.cc:2369] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:20:49.454166 28647 tablet.cc:2369] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:20:49.457497 28239 tablet.cc:2369] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:20:49.533232 28680 raft_consensus.cc:482] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:20:49.533556 28680 raft_consensus.cc:504] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.535059 28680 leader_election.cc:288] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49578faacad8494098af1dfdf9e8837f (127.12.13.2:41827), 0f95737ff9164601ada9728ec4241a4f (127.12.13.3:41011)
I20260808 08:20:49.540212 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.535567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:46569 (local address 127.12.13.3:41011)
0808 08:20:49.535717 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.535722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.535977 (+ 255us) server_negotiation.cc:407] Connection header received
0808 08:20:49.536163 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.536167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.536222 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.536320 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:49.536616 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.537155 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.538917 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.539564 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.539696 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.539765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.539987 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.540063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 08:20:49.540212 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.535429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:37005 (local address 127.12.13.2:41827)
0808 08:20:49.535591 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.535596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.535912 (+ 316us) server_negotiation.cc:407] Connection header received
0808 08:20:49.536099 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.536102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.536144 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.536229 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:20:49.536541 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.537045 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.538917 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.539627 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.539714 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.539765 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.539990 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.540065 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:20:49.540295 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.535389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:41011 (local address 127.12.13.1:46569)
0808 08:20:49.535865 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.535879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:49.536042 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:49.536345 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:49.536348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:49.536367 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:49.536499 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.536504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.537317 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.537320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.538777 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.538789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.539704 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.539710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.539818 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.539877 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 08:20:49.539945 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 08:20:49.540085 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":76,"thread_start_us":134,"threads_started":1}
I20260808 08:20:49.540295 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.535338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:41827 (local address 127.12.13.1:37005)
0808 08:20:49.535806 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.535818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:20:49.535981 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:20:49.536240 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:20:49.536242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:20:49.536258 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:20:49.536424 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.536431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.537155 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.537159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.538777 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:20:49.538789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.539736 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.539739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:20:49.539822 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.539858 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 08:20:49.539944 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 08:20:49.540085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":184,"thread_start_us":196,"threads_started":1}
I20260808 08:20:49.540985 28581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35541fe6efc447b89ca78687bffae77f" candidate_uuid: "b139c45c157a4ee48b4028840efd5a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f95737ff9164601ada9728ec4241a4f" is_pre_election: true
I20260808 08:20:49.541129 28363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35541fe6efc447b89ca78687bffae77f" candidate_uuid: "b139c45c157a4ee48b4028840efd5a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49578faacad8494098af1dfdf9e8837f" is_pre_election: true
I20260808 08:20:49.541365 28581 raft_consensus.cc:2454] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b139c45c157a4ee48b4028840efd5a9b in term 0.
I20260808 08:20:49.541462 28363 raft_consensus.cc:2454] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b139c45c157a4ee48b4028840efd5a9b in term 0.
I20260808 08:20:49.541862 28047 leader_election.cc:302] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f95737ff9164601ada9728ec4241a4f, b139c45c157a4ee48b4028840efd5a9b; no voters:
I20260808 08:20:49.542213 28680 raft_consensus.cc:2781] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:20:49.542294 28680 raft_consensus.cc:482] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:20:49.542333 28680 raft_consensus.cc:3037] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:49.543388 28680 raft_consensus.cc:504] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.543797 28680 leader_election.cc:288] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [CANDIDATE]: Term 1 election: Requested vote from peers 49578faacad8494098af1dfdf9e8837f (127.12.13.2:41827), 0f95737ff9164601ada9728ec4241a4f (127.12.13.3:41011)
I20260808 08:20:49.544169 28363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35541fe6efc447b89ca78687bffae77f" candidate_uuid: "b139c45c157a4ee48b4028840efd5a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49578faacad8494098af1dfdf9e8837f"
I20260808 08:20:49.544250 28581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35541fe6efc447b89ca78687bffae77f" candidate_uuid: "b139c45c157a4ee48b4028840efd5a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f95737ff9164601ada9728ec4241a4f"
I20260808 08:20:49.544296 28363 raft_consensus.cc:3037] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:49.544401 28581 raft_consensus.cc:3037] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:20:49.545518 28363 raft_consensus.cc:2454] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b139c45c157a4ee48b4028840efd5a9b in term 1.
I20260808 08:20:49.545827 28046 leader_election.cc:302] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49578faacad8494098af1dfdf9e8837f, b139c45c157a4ee48b4028840efd5a9b; no voters:
I20260808 08:20:49.545912 28581 raft_consensus.cc:2454] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b139c45c157a4ee48b4028840efd5a9b in term 1.
I20260808 08:20:49.546062 28680 raft_consensus.cc:2781] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:20:49.546315 28680 raft_consensus.cc:686] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 1 LEADER]: Becoming Leader. State: Replica: b139c45c157a4ee48b4028840efd5a9b, State: Running, Role: LEADER
I20260808 08:20:49.546625 28680 consensus_queue.cc:237] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } }
I20260808 08:20:49.549837 27919 catalog_manager.cc:5495] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b reported cstate change: term changed from 0 to 1, leader changed from <none> to b139c45c157a4ee48b4028840efd5a9b (127.12.13.1). New cstate: current_term: 1 leader_uuid: "b139c45c157a4ee48b4028840efd5a9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b139c45c157a4ee48b4028840efd5a9b" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260808 08:20:49.627960 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.622818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.12.13.1:46261)
0808 08:20:49.622961 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 08:20:49.622965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:20:49.623147 (+ 182us) server_negotiation.cc:407] Connection header received
0808 08:20:49.623330 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:20:49.623332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:20:49.623423 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:20:49.623561 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:20:49.623911 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.624792 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.626182 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:20:49.626429 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:20:49.626547 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:20:49.626972 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:20:49.627519 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:20:49.627667 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 08:20:49.627715 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 08:20:49.627787 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:20:49.636173 28581 raft_consensus.cc:1261] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 1 FOLLOWER]: Refusing update from remote peer b139c45c157a4ee48b4028840efd5a9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:49.636418 28363 raft_consensus.cc:1261] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 1 FOLLOWER]: Refusing update from remote peer b139c45c157a4ee48b4028840efd5a9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:20:49.637033 28680 consensus_queue.cc:1035] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f95737ff9164601ada9728ec4241a4f" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:49.637288 28690 consensus_queue.cc:1035] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49578faacad8494098af1dfdf9e8837f" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 41827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:20:49.645504 28694 mvcc.cc:205] Tried to move back new op lower bound from 7316182014500007936 to 7316182014145314816. Current Snapshot: MvccSnapshot[applied={T|T < 7316182014500007936}]
I20260808 08:20:49.654867 27920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:49.655136 27920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:20:49.657811 27920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c5832353c04b94baebc213010751db: Sending DeleteTablet for 3 replicas of tablet 35541fe6efc447b89ca78687bffae77f
I20260808 08:20:49.658727 28136 tablet_service.cc:1514] Processing DeleteTablet for tablet 35541fe6efc447b89ca78687bffae77f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:54000
I20260808 08:20:49.658785 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet 35541fe6efc447b89ca78687bffae77f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:33104
I20260808 08:20:49.659298 28713 tablet_replica.cc:330] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: stopping tablet replica
I20260808 08:20:49.659498 28710 tablet_replica.cc:330] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b: stopping tablet replica
I20260808 08:20:49.659673 28713 raft_consensus.cc:2229] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:49.659904 28710 raft_consensus.cc:2229] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:20:49.660049 28713 raft_consensus.cc:2258] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:49.660108 28551 tablet_service.cc:1514] Processing DeleteTablet for tablet 35541fe6efc447b89ca78687bffae77f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:20:49 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:41384
I20260808 08:20:49.660575 28710 raft_consensus.cc:2258] T 35541fe6efc447b89ca78687bffae77f P b139c45c157a4ee48b4028840efd5a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:49.660598 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 28018
I20260808 08:20:49.668141 28713 ts_tablet_manager.cc:1905] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:49.669639 28714 tablet_replica.cc:330] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: stopping tablet replica
I20260808 08:20:49.670099 28714 raft_consensus.cc:2229] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:20:49.670465 28714 raft_consensus.cc:2258] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:20:49.671763 28713 ts_tablet_manager.cc:1918] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:49.671888 28713 log.cc:1196] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/35541fe6efc447b89ca78687bffae77f
I20260808 08:20:49.672043 28714 ts_tablet_manager.cc:1905] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:20:49.672247 28713 ts_tablet_manager.cc:1939] T 35541fe6efc447b89ca78687bffae77f P 49578faacad8494098af1dfdf9e8837f: Deleting consensus metadata
I20260808 08:20:49.673823 27896 catalog_manager.cc:4841] TS 49578faacad8494098af1dfdf9e8837f (127.12.13.2:41827): tablet 35541fe6efc447b89ca78687bffae77f (table test-table [id=ddaa14d5ce834881a559a85211172f51]) successfully deleted
I20260808 08:20:49.674717 28714 ts_tablet_manager.cc:1918] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:20:49.674842 28714 log.cc:1196] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/35541fe6efc447b89ca78687bffae77f
I20260808 08:20:49.675148 28714 ts_tablet_manager.cc:1939] T 35541fe6efc447b89ca78687bffae77f P 0f95737ff9164601ada9728ec4241a4f: Deleting consensus metadata
I20260808 08:20:49.676352 27898 catalog_manager.cc:4841] TS 0f95737ff9164601ada9728ec4241a4f (127.12.13.3:41011): tablet 35541fe6efc447b89ca78687bffae77f (table test-table [id=ddaa14d5ce834881a559a85211172f51]) successfully deleted
W20260808 08:20:49.677814 27893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:46261 (error 108)
I20260808 08:20:49.678032 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 28245
I20260808 08:20:49.679325 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:20:49.678871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46261 (local address 127.0.0.1:54024)
0808 08:20:49.679108 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 08:20:49.679206 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260808 08:20:49.679565 27893 catalog_manager.cc:4584] TS b139c45c157a4ee48b4028840efd5a9b (127.12.13.1:46261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35541fe6efc447b89ca78687bffae77f: Network error: Client connection negotiation failed: client connection to 127.12.13.1:46261: connect: Connection refused (error 111)
I20260808 08:20:49.692868 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 28440
I20260808 08:20:49.706743 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 27868
2026-08-08T08:20:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2910 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:20:49 dist-test-slave-6ntk krb5kdc[28724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:20:49 dist-test-slave-6ntk krb5kdc[28724](info): set up 2 sockets
Aug 08 08:20:49 dist-test-slave-6ntk krb5kdc[28724](info): commencing operation
krb5kdc: starting...
W20260808 08:20:50.970186 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:20:51 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:20:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/rangeradmin_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/rangeradmin_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/rangerlookup_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/rangerlookup_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
I20260808 08:20:51.169703 12340 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpcRZlg/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/postgres -l logfile start
2026-08-08 08:20:54.025 UTC [28766] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 08:20:54.025 UTC [28766] LOG: listening on IPv4 address "127.12.13.20", port 41929
2026-08-08 08:20:54.029 UTC [28766] LOG: listening on Unix socket "/tmp/.s.PGSQL.41929"
2026-08-08 08:20:54.070 UTC [28770] LOG: database system was shut down at 2026-08-08 08:20:52 UTC
2026-08-08 08:20:54.075 UTC [28766] LOG: database system is ready to accept connections
I20260808 08:20:55.237277 12340 mini_postgres.cc:96] Postgres bound to 41929
2026-08-08 08:20:55.246 UTC [28782] FATAL: database "slave" does not exist
127.12.13.20:41929 - accepting connections
I20260808 08:20:55.246934 12340 mini_ranger.cc:161] Starting Ranger...
I20260808 08:20:55.264047 12340 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 08:20:55.811256 12340 mini_ranger.cc:87] Created ranger Postgres database
I20260808 08:20:55.811388 12340 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-admin
2026-08-08 08:20:56,162 [I] DB FLAVOR :POSTGRES
2026-08-08 08:20:56,162 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 08:20:56,163 [I] Checking connection..
2026-08-08 08:20:56,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 08:20:56,418 [I] Checking connection passed.
2026-08-08 08:20:56,418 [I] --------- Verifying version history table ---------
2026-08-08 08:20:56,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 08:20:56,680 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 08:20:56,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 08:20:56,955 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 08:20:56,955 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 08:20:56,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 08:20:57.222 UTC [28875] WARNING: there is no transaction in progress
2026-08-08 08:20:57,232 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 08:20:57,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 08:20:57,501 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 08:20:57,501 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 08:20:57,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 08:20:57,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 08:20:58,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6ntk.c.gcp-upstream.internal','N') ;"
2026-08-08 08:20:58,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 08:20:58,496 [I] Table x_portal_user does not exist in database ranger
2026-08-08 08:20:58,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 08:20:58,770 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 08:20:58,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 08:20:59,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 08:20:59,280 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 08:20:59,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 08:21:00.343 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.362 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.381 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.399 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.416 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.439 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.507 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.514 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.530 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.542 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.553 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.568 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.579 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.585 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.593 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:00.600 UTC [29038] WARNING: there is no transaction in progress
2026-08-08 08:21:01,326 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 08:21:01,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 08:21:01,585 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 08:21:01,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 08:21:01,857 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 08:21:01,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 08:21:02,129 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 08:21:02,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 08:21:02,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 08:21:02,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6ntk.c.gcp-upstream.internal';"
2026-08-08 08:21:02,908 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 08:21:02,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 08:21:03,189 [I] DB_PATCHES have already been applied
I20260808 08:21:03.195413 12340 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpcRZlg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 08:21:03.195488 12340 mini_ranger.cc:193] Using host: 127.12.13.20
I20260808 08:21:03.200428 12340 mini_ranger.cc:227] Ranger admin URL: http://127.12.13.20:57789
Aug 08, 2026 8:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 8:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 8:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 8:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 8:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 8:21:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 8:21:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 08:21:04.572981 12340 mini_ranger.cc:160] Time spent starting Ranger: real 9.326s user 0.000s sys 0.013s
Aug 08, 2026 8:21:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 8:21:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 08, 2026 8:21:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 8:21:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 8:21:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 08:21:26.230140 12340 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/rangerkms_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/rangerkms_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.13.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.13.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 08:21:26 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177286, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 08:21:26.338387 12340 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 08:21:26.350013 12340 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 08:21:27.336648 12340 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 08:21:27.336772 12340 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms
I20260808 08:21:27.336896 12340 mini_ranger_kms.cc:227] Using postgres at 127.12.13.20:41929
2026-08-08 08:21:27,587 [I] DB FLAVOR :POSTGRES
2026-08-08 08:21:27,587 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 08:21:27,587 [I] Checking connection
2026-08-08 08:21:27,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 08:21:27,851 [I] connection success
2026-08-08 08:21:27,851 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 08:21:27,851 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 08:21:27,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 08:21:28,127 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 08:21:28,128 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 08:21:28,128 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 08:21:28,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:41929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 08:21:28,399 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 08:21:28.860915 12340 mini_ranger_kms.cc:310] Created kms service
I20260808 08:21:29.054651 12340 mini_ranger_kms.cc:326] Created kudu user
I20260808 08:21:29.100793 12340 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 08:21:29 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177286, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:29.516224 12340 mini_ranger_kms.cc:384] Added ranger policy
I20260808 08:21:29.516400 12340 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 08:21:29.516431 12340 mini_ranger_kms.cc:240] Using host: 127.12.13.20
I20260808 08:21:29.521426 12340 mini_ranger_kms.cc:276] Ranger KMS PID: 29317
I20260808 08:21:29.521596 12340 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.13.20:41511
Aug 08, 2026 8:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 8:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 8:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 8:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 8:21:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 8:21:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 08:21:32 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786177292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.13.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:21:32.302805 12340 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.964s user 0.008s sys 0.228s
I20260808 08:21:32.302958 12340 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 08:21:32.303092 12340 mini_ranger_kms.cc:401] 127.12.13.20:41511/kms/v1/keys
Aug 08 08:21:32 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.13.20@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
Aug 08 08:21:33 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.13.20@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
Aug 08 08:21:36 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:21:36.502523 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35023
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:35023
--ranger_config_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:41511/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:21:36.654120 29399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:21:36.654444 29399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:21:36.654487 29399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:21:36.658663 29399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:21:36.658733 29399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:21:36.658752 29399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:21:36.658769 29399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:21:36.658833 29399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:21:36.664803 29399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.13.20:41511/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:35023
--ranger_config_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:21:36.666076 29399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:21:36.667184 29399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:21:36.677982 29405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:21:36.678035 29407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:21:36.678051 29404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:21:36.678658 29399 server_base.cc:1034] running on GCE node
I20260808 08:21:36.679252 29399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:21:36.680644 29399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:21:36.681830 29399 hybrid_clock.cc:648] HybridClock initialized: now 1786177296681808 us; error 40 us; skew 500 ppm
Aug 08 08:21:36 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177296, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:21:36.685405 29399 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:21:36.686628 29399 webserver.cc:460] Webserver started at http://127.12.13.62:43291/ using document root <none> and password file <none>
I20260808 08:21:36.687234 29399 fs_manager.cc:359] Metadata directory not provided
I20260808 08:21:36.687323 29399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:21:36.687570 29399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:21:36 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177296, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:36.740653 29399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "f727fd7d3cec482da68a50058a6b4046"
format_stamp: "Formatted at 2026-08-08 08:21:36 on dist-test-slave-6ntk"
server_key: "120ba49ff59bee8f968777836112d035"
server_key_iv: "ce8ba631edb5e85108cc7ed575cb0532"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:36.741580 29399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "f727fd7d3cec482da68a50058a6b4046"
format_stamp: "Formatted at 2026-08-08 08:21:36 on dist-test-slave-6ntk"
server_key: "120ba49ff59bee8f968777836112d035"
server_key_iv: "ce8ba631edb5e85108cc7ed575cb0532"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:36.746639 29399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260808 08:21:36.765767 29417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:36.767328 29399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:21:36.767465 29399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "f727fd7d3cec482da68a50058a6b4046"
format_stamp: "Formatted at 2026-08-08 08:21:36 on dist-test-slave-6ntk"
server_key: "120ba49ff59bee8f968777836112d035"
server_key_iv: "ce8ba631edb5e85108cc7ed575cb0532"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:36.767637 29399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:21:36.786474 29399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:21:36.789829 29399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:21:36.790143 29399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:21:36.803742 29399 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:35023
I20260808 08:21:36.803799 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:35023 every 8 connection(s)
I20260808 08:21:36.805089 29399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:21:36.808386 29505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:21:36.812933 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 29399
I20260808 08:21:36.813086 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:21:36.813428 12340 external_mini_cluster.cc:1428] Setting key 38218eb5dfb1c4a5bcad5da94b38fa1f
I20260808 08:21:36.815793 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046: Bootstrap starting.
I20260808 08:21:36.819000 29505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046: Neither blocks nor log segments found. Creating new log.
I20260808 08:21:36.820152 29505 log.cc:824] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046: Log is configured to *not* fsync() on all Append() calls
I20260808 08:21:36.822676 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046: No bootstrap required, opened a new log
Aug 08 08:21:36 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:21:36.828464 29505 raft_consensus.cc:348] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f727fd7d3cec482da68a50058a6b4046" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35023 } }
I20260808 08:21:36.828800 29505 raft_consensus.cc:374] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:21:36.828884 29505 raft_consensus.cc:729] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f727fd7d3cec482da68a50058a6b4046, State: Initialized, Role: FOLLOWER
I20260808 08:21:36.829457 29505 consensus_queue.cc:260] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f727fd7d3cec482da68a50058a6b4046" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35023 } }
I20260808 08:21:36.829672 29505 raft_consensus.cc:388] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:21:36.829783 29505 raft_consensus.cc:482] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:21:36.829922 29505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:21:36.831408 29505 raft_consensus.cc:504] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f727fd7d3cec482da68a50058a6b4046" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35023 } }
I20260808 08:21:36.831810 29505 leader_election.cc:302] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f727fd7d3cec482da68a50058a6b4046; no voters:
I20260808 08:21:36.832177 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:36.815310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.12.13.62:35023)
0808 08:21:36.815917 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:36.815943 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:36.815988 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:21:36.817093 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:36.817122 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:36.817629 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:36.818042 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:21:36.818938 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:36.820617 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:36.821799 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:36.822555 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:36.822721 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:36.826145 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:21:36.826156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:21:36.826178 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:21:36.826215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:21:36.828475 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:36.829026 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:36.829031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:36.829038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:36.829177 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:36.829532 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:36.829535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:36.829537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:36.830210 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:21:36.830498 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:36.830727 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 08:21:36.831056 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":173,"threads_started":1}
I20260808 08:21:36.832363 29515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:21:36.832190 29505 leader_election.cc:288] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:21:36.833706 29515 raft_consensus.cc:686] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [term 1 LEADER]: Becoming Leader. State: Replica: f727fd7d3cec482da68a50058a6b4046, State: Running, Role: LEADER
I20260808 08:21:36.834146 29515 consensus_queue.cc:237] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f727fd7d3cec482da68a50058a6b4046" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35023 } }
I20260808 08:21:36.834272 29505 sys_catalog.cc:564] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:21:36.836519 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f727fd7d3cec482da68a50058a6b4046. Latest consensus state: current_term: 1 leader_uuid: "f727fd7d3cec482da68a50058a6b4046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f727fd7d3cec482da68a50058a6b4046" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35023 } } }
I20260808 08:21:36.836561 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f727fd7d3cec482da68a50058a6b4046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f727fd7d3cec482da68a50058a6b4046" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35023 } } }
I20260808 08:21:36.836796 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [sys.catalog]: This master's current role is: LEADER
I20260808 08:21:36.836824 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046 [sys.catalog]: This master's current role is: LEADER
I20260808 08:21:36.837399 29531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:21:36.837944 29505 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 08:21:36.841161 29531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:21:36.858332 29531 catalog_manager.cc:1347] Generated new cluster ID: 4a36d3377de24d8aadeb300fa92fa851
I20260808 08:21:36.858516 29531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:21:36.888542 29531 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:21:36.890769 29531 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:21:36.901532 29531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046: Generated new TSK 0
I20260808 08:21:36.903016 29531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:21:37.866564 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:41511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35023
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:21:38.021692 29569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:21:38.022050 29569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:21:38.022131 29569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:21:38.026494 29569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:21:38.026584 29569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:21:38.026678 29569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:21:38.033497 29569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.13.20:41511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:21:38.035048 29569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:21:38.036422 29569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:21:38.049749 29576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:21:38.049774 29579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:21:38.049959 29569 server_base.cc:1034] running on GCE node
W20260808 08:21:38.049749 29577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:21:38.050707 29569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:21:38.051406 29569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:21:38.052637 29569 hybrid_clock.cc:648] HybridClock initialized: now 1786177298052606 us; error 48 us; skew 500 ppm
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:21:38.056475 29569 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:21:38.057420 29569 webserver.cc:460] Webserver started at http://127.12.13.1:36037/ using document root <none> and password file <none>
I20260808 08:21:38.058115 29569 fs_manager.cc:359] Metadata directory not provided
I20260808 08:21:38.058215 29569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:21:38.058451 29569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:38.077561 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "dc7dea1db45b453c835ac0851d720393"
format_stamp: "Formatted at 2026-08-08 08:21:38 on dist-test-slave-6ntk"
server_key: "962acad0cf64e6c3daeed37129346cea"
server_key_iv: "e26216f91aee0ef1f12039ed78b5c5be"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:38.078392 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "dc7dea1db45b453c835ac0851d720393"
format_stamp: "Formatted at 2026-08-08 08:21:38 on dist-test-slave-6ntk"
server_key: "962acad0cf64e6c3daeed37129346cea"
server_key_iv: "e26216f91aee0ef1f12039ed78b5c5be"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:38.089639 29569 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260808 08:21:38.108623 29592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:38.110562 29569 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260808 08:21:38.110745 29569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "dc7dea1db45b453c835ac0851d720393"
format_stamp: "Formatted at 2026-08-08 08:21:38 on dist-test-slave-6ntk"
server_key: "962acad0cf64e6c3daeed37129346cea"
server_key_iv: "e26216f91aee0ef1f12039ed78b5c5be"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:38.110936 29569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:21:38.146917 29569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:21:38.151584 29569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:21:38.151865 29569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:21:38.152835 29569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:21:38.154120 29569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:21:38.154196 29569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:38.154269 29569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:21:38.154306 29569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:38.177198 29569 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:34081
I20260808 08:21:38.177228 29761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:34081 every 8 connection(s)
I20260808 08:21:38.178843 29569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:21:38.189360 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 29569
I20260808 08:21:38.189638 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:21:38.190233 12340 external_mini_cluster.cc:1428] Setting key bc00e0fae54ecce9f0c4f95b031e46c0
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:21:38.224319 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:38.181468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:51445 (local address 127.12.13.62:35023)
0808 08:21:38.181729 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:38.181732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:38.185275 (+ 3543us) server_negotiation.cc:407] Connection header received
0808 08:21:38.185345 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:38.185348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:38.185435 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:38.185586 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:21:38.188252 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.192144 (+ 3892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.200004 (+ 7860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.200316 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.200468 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:38.218147 (+ 17679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:21:38.218153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:21:38.218156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:21:38.218203 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:21:38.220950 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:38.222062 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:38.222066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:38.222069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:38.222187 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:38.222700 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:38.222704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:38.222706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:38.222989 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:21:38.223082 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:38.223958 (+ 876us) server_negotiation.cc:299] Negotiation successful
0808 08:21:38.224114 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260808 08:21:38.226358 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:38.182063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35023 (local address 127.12.13.1:51445)
0808 08:21:38.182852 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:38.182901 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:38.184871 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:38.186114 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:38.186127 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:38.186874 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:21:38.187919 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:38.187946 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.192507 (+ 4561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.192513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:38.199812 (+ 7299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:38.199830 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.200546 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.200551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:38.200814 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:38.206240 (+ 5426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:21:38.206286 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:21:38.209608 (+ 3322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:21:38.221145 (+ 11537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:21:38.221154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:21:38.221179 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:21:38.221792 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:21:38.222332 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:21:38.222336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:21:38.222338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:21:38.222571 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:21:38.223162 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:21:38.223171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:21:38.223608 (+ 437us) client_negotiation.cc:769] Sending connection context
0808 08:21:38.224593 (+ 985us) client_negotiation.cc:240] Negotiation successful
0808 08:21:38.225034 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":280,"threads_started":1}
I20260808 08:21:38.227957 29762 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:21:38.228229 29762 heartbeater.cc:464] Registering TS with master...
I20260808 08:21:38.228866 29762 heartbeater.cc:511] Master 127.12.13.62:35023 requested a full tablet report, sending...
I20260808 08:21:38.230772 29430 ts_manager.cc:194] Registered new tserver with Master: dc7dea1db45b453c835ac0851d720393 (127.12.13.1:34081)
I20260808 08:21:38.232455 29430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:51445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:21:38.262669 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:41511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35023
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 08:21:38.472301 29774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:21:38.472772 29774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:21:38.472864 29774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:21:38.479151 29774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:21:38.479238 29774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:21:38.479327 29774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:21:38.486907 29774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.13.20:41511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:21:38.488252 29774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:21:38.489457 29774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:21:38.501561 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:21:38.501601 29790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:21:38.502208 29783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:21:38.503602 29774 server_base.cc:1034] running on GCE node
I20260808 08:21:38.504268 29774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:21:38.505293 29774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:21:38.506719 29774 hybrid_clock.cc:648] HybridClock initialized: now 1786177298506680 us; error 62 us; skew 500 ppm
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:21:38.511677 29774 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:21:38.513044 29774 webserver.cc:460] Webserver started at http://127.12.13.2:34533/ using document root <none> and password file <none>
I20260808 08:21:38.514153 29774 fs_manager.cc:359] Metadata directory not provided
I20260808 08:21:38.514267 29774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:21:38.514546 29774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:38.533001 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "6f66e78cfbe546f2be9a91e7a0db344d"
format_stamp: "Formatted at 2026-08-08 08:21:38 on dist-test-slave-6ntk"
server_key: "69edd70536057cfc3c40ddd78269eb59"
server_key_iv: "e1e93bb6baa3ed04421b663138e5b59f"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:38.533813 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "6f66e78cfbe546f2be9a91e7a0db344d"
format_stamp: "Formatted at 2026-08-08 08:21:38 on dist-test-slave-6ntk"
server_key: "69edd70536057cfc3c40ddd78269eb59"
server_key_iv: "e1e93bb6baa3ed04421b663138e5b59f"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:38.539319 29774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 08:21:38.557397 29799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:38.559309 29774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 08:21:38.559478 29774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "6f66e78cfbe546f2be9a91e7a0db344d"
format_stamp: "Formatted at 2026-08-08 08:21:38 on dist-test-slave-6ntk"
server_key: "69edd70536057cfc3c40ddd78269eb59"
server_key_iv: "e1e93bb6baa3ed04421b663138e5b59f"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:38.559633 29774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:21:38.599601 29774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:21:38.604799 29774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:21:38.605110 29774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:21:38.606159 29774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:21:38.607713 29774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:21:38.607831 29774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:38.607936 29774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:21:38.607986 29774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:38.634624 29774 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:35589
I20260808 08:21:38.636422 29774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:21:38.638000 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 29774
I20260808 08:21:38.638155 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:21:38.638557 12340 external_mini_cluster.cc:1428] Setting key 43c7fd2f1c2f56d6166af7fda843c173
I20260808 08:21:38.646328 29975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:35589 every 8 connection(s)
Aug 08 08:21:38 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:21:38.688225 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:38.654024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:36585 (local address 127.12.13.62:35023)
0808 08:21:38.654263 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:38.654272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:38.662591 (+ 8319us) server_negotiation.cc:407] Connection header received
0808 08:21:38.664335 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:38.664340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:38.664403 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:38.664463 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:21:38.670101 (+ 5638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.671020 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.675670 (+ 4650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.676103 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.676237 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:38.682411 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:21:38.682416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:21:38.682419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:21:38.682458 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:21:38.684656 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:38.685533 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:38.685538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:38.685539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:38.685679 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:38.686312 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:38.686316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:38.686318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:38.686629 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:21:38.686745 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:38.687792 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0808 08:21:38.687999 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:21:38.691016 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:38.652660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35023 (local address 127.12.13.2:36585)
0808 08:21:38.662352 (+ 9692us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:38.662442 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:38.664025 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:38.664954 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:38.664967 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:38.665473 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:21:38.667450 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:38.667480 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.671229 (+ 3749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.671233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:38.673361 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:38.673376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:38.676940 (+ 3564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:38.676944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:38.677230 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:38.678778 (+ 1548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:21:38.678805 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:21:38.682115 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:21:38.684826 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:21:38.684833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:21:38.684854 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:21:38.685323 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:21:38.685838 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:21:38.685843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:21:38.685847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:21:38.686131 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:21:38.686840 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:21:38.686847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:21:38.687415 (+ 568us) client_negotiation.cc:769] Sending connection context
0808 08:21:38.688916 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0808 08:21:38.689371 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9354,"thread_start_us":239,"threads_started":1}
I20260808 08:21:38.693152 29976 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35023
I20260808 08:21:38.693463 29976 heartbeater.cc:464] Registering TS with master...
I20260808 08:21:38.694300 29976 heartbeater.cc:511] Master 127.12.13.62:35023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:21:38.695670 29430 ts_manager.cc:194] Registered new tserver with Master: 6f66e78cfbe546f2be9a91e7a0db344d (127.12.13.2:35589)
I20260808 08:21:38.696539 29430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:36585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
W20260808 08:21:38.741796 12340 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.13.3: real 0.101s user 0.004s sys 0.003s
I20260808 08:21:38.742436 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:41511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35023
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:21:38.949832 29985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:21:38.950788 29985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:21:38.950922 29985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:21:38.957175 29985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:21:38.957455 29985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:21:38.957625 29985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:21:38.965803 29985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.13.20:41511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:21:38.967864 29985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:21:38.969581 29985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:21:38.986979 29992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:21:38.987403 29985 server_base.cc:1034] running on GCE node
W20260808 08:21:38.988037 29991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:21:38.992846 29994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:21:38.993798 29985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:21:38.995049 29985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:21:39.010610 29985 hybrid_clock.cc:648] HybridClock initialized: now 1786177299010496 us; error 99 us; skew 500 ppm
Aug 08 08:21:39 dist-test-slave-6ntk krb5kdc[28724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:21:39.016513 29985 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
Aug 08 08:21:39 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:39.023590 29985 webserver.cc:460] Webserver started at http://127.12.13.3:42039/ using document root <none> and password file <none>
I20260808 08:21:39.024506 29985 fs_manager.cc:359] Metadata directory not provided
I20260808 08:21:39.024657 29985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:21:39.024988 29985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:21:39 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:39.048930 29985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62"
format_stamp: "Formatted at 2026-08-08 08:21:39 on dist-test-slave-6ntk"
server_key: "71c618c505465615ce0fa3a895e8e439"
server_key_iv: "efd88d06c1cf6f8c3cbcdca5df3f49b7"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:39.049983 29985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62"
format_stamp: "Formatted at 2026-08-08 08:21:39 on dist-test-slave-6ntk"
server_key: "71c618c505465615ce0fa3a895e8e439"
server_key_iv: "efd88d06c1cf6f8c3cbcdca5df3f49b7"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:39.066656 29985 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.002s sys 0.003s
I20260808 08:21:39.096601 30005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:39.099192 29985 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260808 08:21:39.099593 29985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62"
format_stamp: "Formatted at 2026-08-08 08:21:39 on dist-test-slave-6ntk"
server_key: "71c618c505465615ce0fa3a895e8e439"
server_key_iv: "efd88d06c1cf6f8c3cbcdca5df3f49b7"
server_key_version: "kuduclusterkey@0"
I20260808 08:21:39.099881 29985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:21:39.120774 29985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:21:39.125430 29985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:21:39.125787 29985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:21:39.126950 29985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:21:39.128336 29985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:21:39.128468 29985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:39.128595 29985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:21:39.128683 29985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:21:39.157044 29985 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:43949
I20260808 08:21:39.159451 30182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:43949 every 8 connection(s)
I20260808 08:21:39.160620 29985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:21:39.170754 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 29985
I20260808 08:21:39.170898 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:21:39.171223 12340 external_mini_cluster.cc:1428] Setting key 5bec32ef2f6c7c3fe4258982bfc2ce13
Aug 08 08:21:39 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:21:39.205401 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.165413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:55455 (local address 127.12.13.62:35023)
0808 08:21:39.165545 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.165549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.177461 (+ 11912us) server_negotiation.cc:407] Connection header received
0808 08:21:39.179214 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.179218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.179280 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.179354 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:21:39.181624 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.182512 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.184370 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.184687 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.184770 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.200574 (+ 15804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:21:39.200579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:21:39.200582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:21:39.200616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:21:39.202667 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:39.203423 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:39.203427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:39.203429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:39.203531 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:39.203984 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:39.203988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:39.203990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:39.204247 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:21:39.204353 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.205071 (+ 718us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.205224 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:21:39.206660 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.164104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:35023 (local address 127.12.13.3:55455)
0808 08:21:39.177223 (+ 13119us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.177324 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.178926 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.179838 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:39.179850 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:39.180484 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:21:39.181417 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.181440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.182691 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.182694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.184192 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.184204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.185001 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.185005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.185481 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.198187 (+ 12706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:21:39.198215 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:21:39.200308 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:21:39.202824 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:21:39.202833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:21:39.202858 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:21:39.203292 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:21:39.203628 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:21:39.203631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:21:39.203633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:21:39.203860 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:21:39.204366 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:21:39.204374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:21:39.204761 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 08:21:39.205017 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 08:21:39.205441 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12720,"thread_start_us":190,"threads_started":1}
I20260808 08:21:39.214319 30183 heartbeater.cc:347] Connected to a master server at 127.12.13.62:35023
I20260808 08:21:39.214723 30183 heartbeater.cc:464] Registering TS with master...
I20260808 08:21:39.215426 30183 heartbeater.cc:511] Master 127.12.13.62:35023 requested a full tablet report, sending...
I20260808 08:21:39.217111 29430 ts_manager.cc:194] Registered new tserver with Master: 97d56f6cd5e14bcb92d5d1d6cba6eb62 (127.12.13.3:43949)
I20260808 08:21:39.217974 29430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:55455
I20260808 08:21:39.230819 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:21:39.235613 29762 heartbeater.cc:503] Master 127.12.13.62:35023 was elected leader, sending a full tablet report...
I20260808 08:21:39.253443 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.241541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.12.13.62:35023)
0808 08:21:39.241673 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.241677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.242640 (+ 963us) server_negotiation.cc:407] Connection header received
0808 08:21:39.242910 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.242914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.242971 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.243028 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:21:39.244321 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.245180 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.246881 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.247234 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.247315 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.249496 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:21:39.249500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:21:39.249503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:21:39.249533 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:21:39.251406 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:39.251956 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:39.251960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:39.251962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:39.252055 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:21:39.252378 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:21:39.252382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:21:39.252384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:21:39.252615 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:21:39.252726 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.253135 (+ 409us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.253264 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:21:39.257514 29430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:21:39.262444 29430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:21:39.294858 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.277431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:35589 (local address 127.0.0.1:58522)
0808 08:21:39.281350 (+ 3919us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.281413 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.282614 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.287778 (+ 5164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:39.287786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:39.287826 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:21:39.288163 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.288169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.289802 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.289806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.292648 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.292661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.294257 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.294261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.294375 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.294416 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 08:21:39.294535 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 08:21:39.294664 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3701,"mutex_wait_us":251,"spinlock_wait_cycles":6784,"thread_start_us":153,"threads_started":1}
I20260808 08:21:39.295426 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.277564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.12.13.2:35589)
0808 08:21:39.286964 (+ 9400us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.286971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.286992 (+ 21us) server_negotiation.cc:407] Connection header received
0808 08:21:39.287067 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.287073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.287325 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.287553 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:21:39.288284 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.289659 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.292842 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.293820 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.294010 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.294206 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.295097 (+ 891us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.295222 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9294,"thread_start_us":149,"threads_started":1}
I20260808 08:21:39.296460 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.280410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34081 (local address 127.0.0.1:54408)
0808 08:21:39.286579 (+ 6169us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.286592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.286690 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.287778 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:39.287786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:39.287821 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:21:39.287986 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.287993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.292877 (+ 4884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.292880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.294071 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.294081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.295646 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.295649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.296148 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.296172 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:21:39.296245 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 08:21:39.296313 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6104,"mutex_wait_us":397,"spinlock_wait_cycles":768,"thread_start_us":140,"threads_started":1}
I20260808 08:21:39.298812 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.280879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.12.13.1:34081)
0808 08:21:39.286940 (+ 6061us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.286949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.286978 (+ 29us) server_negotiation.cc:407] Connection header received
0808 08:21:39.287068 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.287073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.287331 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.287569 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:21:39.291524 (+ 3955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.292751 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.294207 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.295268 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.295385 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.295573 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.298502 (+ 2929us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.298608 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5930,"thread_start_us":248,"threads_started":1}
I20260808 08:21:39.299069 29882 tablet_service.cc:1467] Processing CreateTablet for tablet 19f0ad343d174051b8ac5bb9d2168f1f (DEFAULT_TABLE table=test-table [id=f82255e8f9f144cb846fb81f7f94ea4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:21:39.300565 29882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f0ad343d174051b8ac5bb9d2168f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:21:39.301966 29660 tablet_service.cc:1467] Processing CreateTablet for tablet 19f0ad343d174051b8ac5bb9d2168f1f (DEFAULT_TABLE table=test-table [id=f82255e8f9f144cb846fb81f7f94ea4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:21:39.303252 29660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f0ad343d174051b8ac5bb9d2168f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:21:39.305534 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.279590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:43949 (local address 127.0.0.1:56682)
0808 08:21:39.286158 (+ 6568us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.286174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.286313 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.295617 (+ 9304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:39.295621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:39.295644 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:21:39.295774 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.295780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.300553 (+ 4773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.300559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.301766 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.301781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.305036 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.305040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.305178 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.305221 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 08:21:39.305283 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 08:21:39.305378 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6490,"mutex_wait_us":3,"thread_start_us":171,"threads_started":1}
I20260808 08:21:39.306433 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.280096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.12.13.3:43949)
0808 08:21:39.290298 (+ 10202us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.294948 (+ 4650us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.295008 (+ 60us) server_negotiation.cc:407] Connection header received
0808 08:21:39.295114 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.295120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.295380 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.297347 (+ 1967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:21:39.297519 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.300397 (+ 2878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.302120 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.304877 (+ 2757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.305821 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.306044 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.306100 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.306200 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10012,"thread_start_us":167,"threads_started":1}
I20260808 08:21:39.310026 30054 tablet_service.cc:1467] Processing CreateTablet for tablet 19f0ad343d174051b8ac5bb9d2168f1f (DEFAULT_TABLE table=test-table [id=f82255e8f9f144cb846fb81f7f94ea4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:21:39.311216 30054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f0ad343d174051b8ac5bb9d2168f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:21:39.314132 30216 tablet_bootstrap.cc:492] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: Bootstrap starting.
I20260808 08:21:39.316668 30215 tablet_bootstrap.cc:492] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Bootstrap starting.
I20260808 08:21:39.317394 30216 tablet_bootstrap.cc:654] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: Neither blocks nor log segments found. Creating new log.
I20260808 08:21:39.318987 30216 log.cc:824] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: Log is configured to *not* fsync() on all Append() calls
I20260808 08:21:39.319950 30215 tablet_bootstrap.cc:654] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Neither blocks nor log segments found. Creating new log.
I20260808 08:21:39.321452 30215 log.cc:824] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:21:39.323707 30216 tablet_bootstrap.cc:492] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: No bootstrap required, opened a new log
I20260808 08:21:39.323799 30218 tablet_bootstrap.cc:492] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Bootstrap starting.
I20260808 08:21:39.324056 30216 ts_tablet_manager.cc:1397] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260808 08:21:39.327399 30218 tablet_bootstrap.cc:654] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Neither blocks nor log segments found. Creating new log.
I20260808 08:21:39.329001 30218 log.cc:824] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Log is configured to *not* fsync() on all Append() calls
I20260808 08:21:39.331687 30216 raft_consensus.cc:348] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.332073 30216 raft_consensus.cc:374] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:21:39.332173 30216 raft_consensus.cc:729] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7dea1db45b453c835ac0851d720393, State: Initialized, Role: FOLLOWER
I20260808 08:21:39.332444 30215 tablet_bootstrap.cc:492] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: No bootstrap required, opened a new log
I20260808 08:21:39.332492 30218 tablet_bootstrap.cc:492] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: No bootstrap required, opened a new log
I20260808 08:21:39.332726 30215 ts_tablet_manager.cc:1397] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20260808 08:21:39.332827 30218 ts_tablet_manager.cc:1397] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 08:21:39.332871 30216 consensus_queue.cc:260] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.334599 30216 ts_tablet_manager.cc:1428] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260808 08:21:39.340915 30215 raft_consensus.cc:348] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.341274 30215 raft_consensus.cc:374] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:21:39.341099 30218 raft_consensus.cc:348] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.341392 30218 raft_consensus.cc:374] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:21:39.341388 30215 raft_consensus.cc:729] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f66e78cfbe546f2be9a91e7a0db344d, State: Initialized, Role: FOLLOWER
I20260808 08:21:39.341451 30218 raft_consensus.cc:729] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d56f6cd5e14bcb92d5d1d6cba6eb62, State: Initialized, Role: FOLLOWER
I20260808 08:21:39.342044 30215 consensus_queue.cc:260] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.342118 30218 consensus_queue.cc:260] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.343592 30215 ts_tablet_manager.cc:1428] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20260808 08:21:39.343618 30218 ts_tablet_manager.cc:1428] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260808 08:21:39.344161 30183 heartbeater.cc:503] Master 127.12.13.62:35023 was elected leader, sending a full tablet report...
I20260808 08:21:39.344375 29976 heartbeater.cc:503] Master 127.12.13.62:35023 was elected leader, sending a full tablet report...
W20260808 08:21:39.402602 29977 tablet.cc:2369] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:21:39.412096 30184 tablet.cc:2369] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:21:39.431782 29764 tablet.cc:2369] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:21:39.475441 30222 raft_consensus.cc:482] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:21:39.475715 30222 raft_consensus.cc:504] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.477237 30222 leader_election.cc:288] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f66e78cfbe546f2be9a91e7a0db344d (127.12.13.2:35589), 97d56f6cd5e14bcb92d5d1d6cba6eb62 (127.12.13.3:43949)
I20260808 08:21:39.493160 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.482324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:47741 (local address 127.12.13.3:43949)
0808 08:21:39.482454 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.482459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.483167 (+ 708us) server_negotiation.cc:407] Connection header received
0808 08:21:39.483395 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.483401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.483456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.483506 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:21:39.485123 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.486217 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.491749 (+ 5532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.492364 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.492459 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.492541 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.492921 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.493008 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 08:21:39.493908 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.481838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:43949 (local address 127.12.13.1:47741)
0808 08:21:39.483058 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.483074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.483259 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.483962 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:39.483965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:39.483989 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:21:39.484382 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.484392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.486542 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.486545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.491563 (+ 5018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.491576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.492646 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.492649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.492773 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.492798 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 08:21:39.493610 (+ 812us) client_negotiation.cc:240] Negotiation successful
0808 08:21:39.493711 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"thread_start_us":124,"threads_started":1}
I20260808 08:21:39.494784 30100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f0ad343d174051b8ac5bb9d2168f1f" candidate_uuid: "dc7dea1db45b453c835ac0851d720393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" is_pre_election: true
I20260808 08:21:39.495178 30100 raft_consensus.cc:2454] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc7dea1db45b453c835ac0851d720393 in term 0.
I20260808 08:21:39.495784 29598 leader_election.cc:302] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d56f6cd5e14bcb92d5d1d6cba6eb62, dc7dea1db45b453c835ac0851d720393; no voters:
I20260808 08:21:39.496156 30222 raft_consensus.cc:2781] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:21:39.496273 30222 raft_consensus.cc:482] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:21:39.496330 30222 raft_consensus.cc:3037] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:21:39.497828 30222 raft_consensus.cc:504] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.498319 30222 leader_election.cc:288] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [CANDIDATE]: Term 1 election: Requested vote from peers 6f66e78cfbe546f2be9a91e7a0db344d (127.12.13.2:35589), 97d56f6cd5e14bcb92d5d1d6cba6eb62 (127.12.13.3:43949)
I20260808 08:21:39.504019 30100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f0ad343d174051b8ac5bb9d2168f1f" candidate_uuid: "dc7dea1db45b453c835ac0851d720393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62"
I20260808 08:21:39.504205 30100 raft_consensus.cc:3037] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:21:39.505841 30100 raft_consensus.cc:2454] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc7dea1db45b453c835ac0851d720393 in term 1.
I20260808 08:21:39.506168 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.478085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56495 (local address 127.12.13.2:35589)
0808 08:21:39.478207 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.478212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.478229 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:21:39.490225 (+ 11996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.490230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.490298 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.490385 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:21:39.501124 (+ 10739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.502765 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.504617 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.505274 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.505362 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.505438 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.505824 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.506030 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:21:39.506433 29598 leader_election.cc:302] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d56f6cd5e14bcb92d5d1d6cba6eb62, dc7dea1db45b453c835ac0851d720393; no voters:
I20260808 08:21:39.506634 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.477485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:35589 (local address 127.12.13.1:56495)
0808 08:21:39.478009 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.478028 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.490064 (+ 12036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.497974 (+ 7910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:21:39.497979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:21:39.498010 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:21:39.498214 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.498220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.503032 (+ 4812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.503036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.504455 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:21:39.504470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.505548 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.505551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:21:39.505674 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.505701 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:21:39.506337 (+ 636us) client_negotiation.cc:240] Negotiation successful
0808 08:21:39.506435 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":168,"threads_started":1}
I20260808 08:21:39.506671 30222 raft_consensus.cc:2781] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:21:39.506911 30222 raft_consensus.cc:686] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [term 1 LEADER]: Becoming Leader. State: Replica: dc7dea1db45b453c835ac0851d720393, State: Running, Role: LEADER
I20260808 08:21:39.507342 30222 consensus_queue.cc:237] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } }
I20260808 08:21:39.509719 29904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f0ad343d174051b8ac5bb9d2168f1f" candidate_uuid: "dc7dea1db45b453c835ac0851d720393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f66e78cfbe546f2be9a91e7a0db344d"
I20260808 08:21:39.509776 29905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f0ad343d174051b8ac5bb9d2168f1f" candidate_uuid: "dc7dea1db45b453c835ac0851d720393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" is_pre_election: true
I20260808 08:21:39.510098 29905 raft_consensus.cc:2454] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc7dea1db45b453c835ac0851d720393 in term 0.
I20260808 08:21:39.513797 29442 catalog_manager.cc:5495] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc7dea1db45b453c835ac0851d720393 (127.12.13.1). New cstate: current_term: 1 leader_uuid: "dc7dea1db45b453c835ac0851d720393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc7dea1db45b453c835ac0851d720393" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 34081 } health_report { overall_health: HEALTHY } } }
Aug 08 08:21:39 dist-test-slave-6ntk krb5kdc[28724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:21:39.583603 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.574062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.12.13.1:34081)
0808 08:21:39.577985 (+ 3923us) server_negotiation.cc:206] Beginning negotiation
0808 08:21:39.577991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:21:39.578011 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:21:39.578117 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:21:39.578120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:21:39.578282 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:21:39.578430 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:21:39.578785 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.579658 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.581097 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:21:39.581480 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:21:39.581584 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:21:39.582100 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:21:39.582734 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:21:39.582866 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 08:21:39.583341 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 08:21:39.583413 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3779}
I20260808 08:21:39.591689 29904 raft_consensus.cc:3037] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:21:39.592379 30100 raft_consensus.cc:1261] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 1 FOLLOWER]: Refusing update from remote peer dc7dea1db45b453c835ac0851d720393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:21:39.593397 29904 raft_consensus.cc:1261] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 1 FOLLOWER]: Refusing update from remote peer dc7dea1db45b453c835ac0851d720393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:21:39.593832 30230 consensus_queue.cc:1035] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d56f6cd5e14bcb92d5d1d6cba6eb62" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 43949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:21:39.594187 30222 consensus_queue.cc:1035] T 19f0ad343d174051b8ac5bb9d2168f1f P dc7dea1db45b453c835ac0851d720393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f66e78cfbe546f2be9a91e7a0db344d" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:21:39.637691 30242 mvcc.cc:205] Tried to move back new op lower bound from 7316182219118387200 to 7316182218785153024. Current Snapshot: MvccSnapshot[applied={T|T < 7316182219118387200}]
I20260808 08:21:39.649487 29442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:21:39.649710 29442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:21:39.652798 29442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f727fd7d3cec482da68a50058a6b4046: Sending DeleteTablet for 3 replicas of tablet 19f0ad343d174051b8ac5bb9d2168f1f
I20260808 08:21:39.654320 29882 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f0ad343d174051b8ac5bb9d2168f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:21:39 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:58522
I20260808 08:21:39.654817 30054 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f0ad343d174051b8ac5bb9d2168f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:21:39 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:56682
I20260808 08:21:39.655251 29660 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f0ad343d174051b8ac5bb9d2168f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:21:39 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:54408
I20260808 08:21:39.655395 30252 tablet_replica.cc:330] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: stopping tablet replica
I20260808 08:21:39.655786 30252 raft_consensus.cc:2229] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:21:39.656144 30252 raft_consensus.cc:2258] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:21:39.656435 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 29569
I20260808 08:21:39.664772 30252 ts_tablet_manager.cc:1905] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:21:39.668022 30252 ts_tablet_manager.cc:1918] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:21:39.668128 30252 log.cc:1196] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/19f0ad343d174051b8ac5bb9d2168f1f
I20260808 08:21:39.668473 30252 ts_tablet_manager.cc:1939] T 19f0ad343d174051b8ac5bb9d2168f1f P 97d56f6cd5e14bcb92d5d1d6cba6eb62: Deleting consensus metadata
I20260808 08:21:39.669801 29420 catalog_manager.cc:4841] TS 97d56f6cd5e14bcb92d5d1d6cba6eb62 (127.12.13.3:43949): tablet 19f0ad343d174051b8ac5bb9d2168f1f (table test-table [id=f82255e8f9f144cb846fb81f7f94ea4d]) successfully deleted
I20260808 08:21:39.674497 30251 tablet_replica.cc:330] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: stopping tablet replica
I20260808 08:21:39.674866 30251 raft_consensus.cc:2229] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:21:39.675239 30251 raft_consensus.cc:2258] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:21:39.679848 30251 ts_tablet_manager.cc:1905] T 19f0ad343d174051b8ac5bb9d2168f1f P 6f66e78cfbe546f2be9a91e7a0db344d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 08:21:39.690100 29420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:34081 (error 108)
I20260808 08:21:39.691525 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 29774
I20260808 08:21:39.691908 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:21:39.691557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:34081 (local address 127.0.0.1:54422)
0808 08:21:39.691758 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.691825 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:34081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260808 08:21:39.695855 29420 catalog_manager.cc:4584] TS dc7dea1db45b453c835ac0851d720393 (127.12.13.1:34081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19f0ad343d174051b8ac5bb9d2168f1f: Network error: Client connection negotiation failed: client connection to 127.12.13.1:34081: connect: Connection refused (error 111)
W20260808 08:21:39.728232 30210 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:21:39.727455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:35589 (local address 127.0.0.1:58524)
0808 08:21:39.727689 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 08:21:39.727708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:21:39.727821 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:21:39.728072 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.2:35589: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129}
I20260808 08:21:39.728972 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 29985
I20260808 08:21:39.748276 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 29399
I20260808 08:21:39.763763 12340 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 08:21:40.120283 12340 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 08:21:40.120455 12340 mini_ranger.cc:66] Stopping Ranger...
I20260808 08:21:40.466156 12340 mini_ranger.cc:68] Stopped Ranger
2026-08-08 08:21:40.466 UTC [28766] LOG: received smart shutdown request
2026-08-08 08:21:40.471 UTC [28766] LOG: background worker "logical replication launcher" (PID 28780) exited with exit code 1
2026-08-08 08:21:40.471 UTC [28771] LOG: shutting down
2026-08-08 08:21:40.660 UTC [28766] LOG: database system is shut down
2026-08-08T08:21:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51069 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:21:40 dist-test-slave-6ntk krb5kdc[30268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:21:40 dist-test-slave-6ntk krb5kdc[30268](info): set up 2 sockets
Aug 08 08:21:40 dist-test-slave-6ntk krb5kdc[30268](info): commencing operation
krb5kdc: starting...
W20260808 08:21:42.087831 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:21:42 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:21:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/rangeradmin_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/rangeradmin_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/rangerlookup_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/rangerlookup_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
I20260808 08:21:42.284955 12340 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpcRZlg/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/postgres -l logfile start
2026-08-08 08:21:45.125 UTC [30308] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 08:21:45.125 UTC [30308] LOG: listening on IPv4 address "127.12.13.20", port 43235
2026-08-08 08:21:45.129 UTC [30308] LOG: listening on Unix socket "/tmp/.s.PGSQL.43235"
2026-08-08 08:21:45.168 UTC [30315] LOG: database system was shut down at 2026-08-08 08:21:43 UTC
2026-08-08 08:21:45.173 UTC [30308] LOG: database system is ready to accept connections
I20260808 08:21:46.346073 12340 mini_postgres.cc:96] Postgres bound to 43235
2026-08-08 08:21:46.354 UTC [30329] FATAL: database "slave" does not exist
127.12.13.20:43235 - accepting connections
I20260808 08:21:46.354913 12340 mini_ranger.cc:161] Starting Ranger...
I20260808 08:21:46.373525 12340 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 08:21:46.944208 12340 mini_ranger.cc:87] Created ranger Postgres database
I20260808 08:21:46.944289 12340 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-admin
2026-08-08 08:21:47,261 [I] DB FLAVOR :POSTGRES
2026-08-08 08:21:47,261 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 08:21:47,261 [I] Checking connection..
2026-08-08 08:21:47,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 08:21:47,534 [I] Checking connection passed.
2026-08-08 08:21:47,534 [I] --------- Verifying version history table ---------
2026-08-08 08:21:47,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 08:21:47,828 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 08:21:47,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 08:21:48,110 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 08:21:48,110 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 08:21:48,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 08:21:48.373 UTC [30432] WARNING: there is no transaction in progress
2026-08-08 08:21:48,382 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 08:21:48,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 08:21:48,653 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 08:21:48,653 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 08:21:48,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 08:21:48,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 08:21:49,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6ntk.c.gcp-upstream.internal','N') ;"
2026-08-08 08:21:49,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 08:21:49,701 [I] Table x_portal_user does not exist in database ranger
2026-08-08 08:21:49,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 08:21:49,985 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 08:21:49,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 08:21:50,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 08:21:50,530 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 08:21:50,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 08:21:51.587 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.604 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.623 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.642 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.662 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.681 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.745 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.754 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.771 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.782 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.794 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.809 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.820 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.828 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.836 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:51.844 UTC [30601] WARNING: there is no transaction in progress
2026-08-08 08:21:52,593 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 08:21:52,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 08:21:52,839 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 08:21:52,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 08:21:53,124 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 08:21:53,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 08:21:53,422 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 08:21:53,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 08:21:53,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 08:21:53,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6ntk.c.gcp-upstream.internal';"
2026-08-08 08:21:54,237 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 08:21:54,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 08:21:54,511 [I] DB_PATCHES have already been applied
I20260808 08:21:54.517122 12340 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpcRZlg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 08:21:54.517189 12340 mini_ranger.cc:193] Using host: 127.12.13.20
I20260808 08:21:54.521807 12340 mini_ranger.cc:227] Ranger admin URL: http://127.12.13.20:35049
Aug 08, 2026 8:21:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 8:21:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 8:21:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 8:21:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 8:21:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 8:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 8:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 08:21:55.872615 12340 mini_ranger.cc:160] Time spent starting Ranger: real 9.518s user 0.000s sys 0.013s
Aug 08, 2026 8:22:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 8:22:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 8:22:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 8:22:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 8:22:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 08:22:19.767016 12340 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/rangerkms_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/rangerkms_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.13.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.13.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.13.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/HTTP_127.12.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 08:22:19 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 08:22:19.869626 12340 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 08:22:19.882002 12340 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 08:22:20.926824 12340 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 08:22:20.926925 12340 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms
I20260808 08:22:20.926952 12340 mini_ranger_kms.cc:227] Using postgres at 127.12.13.20:43235
2026-08-08 08:22:21,124 [I] DB FLAVOR :POSTGRES
2026-08-08 08:22:21,124 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 08:22:21,124 [I] Checking connection
2026-08-08 08:22:21,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 08:22:21,406 [I] connection success
2026-08-08 08:22:21,407 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 08:22:21,407 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 08:22:21,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 08:22:21,677 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 08:22:21,678 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 08:22:21,678 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 08:22:21,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.13.20:43235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 08:22:21,953 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 08:22:22.510603 12340 mini_ranger_kms.cc:310] Created kms service
I20260808 08:22:22.711115 12340 mini_ranger_kms.cc:326] Created kudu user
I20260808 08:22:22.762259 12340 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 08:22:22 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:23.202296 12340 mini_ranger_kms.cc:384] Added ranger policy
I20260808 08:22:23.202492 12340 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpcRZlg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpcRZlg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 08:22:23.202515 12340 mini_ranger_kms.cc:240] Using host: 127.12.13.20
I20260808 08:22:23.207546 12340 mini_ranger_kms.cc:276] Ranger KMS PID: 30884
I20260808 08:22:23.207644 12340 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.13.20:50517
Aug 08, 2026 8:22:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 8:22:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 8:22:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 8:22:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 8:22:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 8:22:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 08:22:25 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786177345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.13.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:25.970144 12340 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.101s user 0.012s sys 0.165s
I20260808 08:22:25.970317 12340 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 08:22:25.970394 12340 mini_ranger_kms.cc:401] 127.12.13.20:50517/kms/v1/keys
Aug 08 08:22:26 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.13.20@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
Aug 08 08:22:26 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.13.20@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
Aug 08 08:22:28 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:22:28.829523 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46183
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:46183
--ranger_config_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:28.955485 30967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:28.955803 30967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:28.955849 30967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:28.960116 30967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:28.960213 30967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:28.960237 30967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:28.960256 30967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:28.960276 30967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:28.966730 30967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.13.20:50517/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:46183
--ranger_config_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:28.968043 30967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:28.969348 30967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:28.979100 30972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:28.979100 30977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:28.979144 30967 server_base.cc:1034] running on GCE node
W20260808 08:22:28.979172 30974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:28.980057 30967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:28.981213 30967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:28.982410 30967 hybrid_clock.cc:648] HybridClock initialized: now 1786177348982390 us; error 45 us; skew 500 ppm
Aug 08 08:22:28 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:28.985450 30967 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:28.986608 30967 webserver.cc:460] Webserver started at http://127.12.13.62:32851/ using document root <none> and password file <none>
I20260808 08:22:28.987263 30967 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:28.987356 30967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:28.987623 30967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:22:28 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:29.046497 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "55cbf1d9b94048fd9cc6e4373c879aab"
format_stamp: "Formatted at 2026-08-08 08:22:29 on dist-test-slave-6ntk"
server_key: "21b5900bead81b558a743d9b56aaa267"
server_key_iv: "fda9b3e23ece8d451e1c76460c2d7a29"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:29.047176 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "55cbf1d9b94048fd9cc6e4373c879aab"
format_stamp: "Formatted at 2026-08-08 08:22:29 on dist-test-slave-6ntk"
server_key: "21b5900bead81b558a743d9b56aaa267"
server_key_iv: "fda9b3e23ece8d451e1c76460c2d7a29"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:29.051800 30967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:22:29.071053 30991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:29.072625 30967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:22:29.072808 30967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "55cbf1d9b94048fd9cc6e4373c879aab"
format_stamp: "Formatted at 2026-08-08 08:22:29 on dist-test-slave-6ntk"
server_key: "21b5900bead81b558a743d9b56aaa267"
server_key_iv: "fda9b3e23ece8d451e1c76460c2d7a29"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:29.072958 30967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:29.083804 30967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:29.087177 30967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:29.087432 30967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:29.096230 30967 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:46183
I20260808 08:22:29.096243 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:46183 every 8 connection(s)
I20260808 08:22:29.097337 30967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:29.099321 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 30967
I20260808 08:22:29.099462 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:22:29.099824 12340 external_mini_cluster.cc:1428] Setting key 0b9fba21c0f2317fa05e17b17c80884d
I20260808 08:22:29.101032 31065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:29.107971 31065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Bootstrap starting.
I20260808 08:22:29.111207 31065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Neither blocks nor log segments found. Creating new log.
Aug 08 08:22:29 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:29.112141 31065 log.cc:824] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:29.114864 31065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: No bootstrap required, opened a new log
I20260808 08:22:29.118216 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:29.101452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.12.13.62:46183)
0808 08:22:29.102014 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:29.102036 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:29.102092 (+ 56us) server_negotiation.cc:407] Connection header received
0808 08:22:29.102986 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:29.103010 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:29.103470 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:29.103855 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:29.104882 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:29.106491 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:29.107717 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:29.108427 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:29.108583 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:29.111950 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:29.111959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:29.111979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:29.112017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:29.114730 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:29.115275 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:29.115282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:29.115288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:29.115425 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:29.115749 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:29.115752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:29.115754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:29.116279 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:29.116488 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:29.116842 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 08:22:29.117145 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":176,"threads_started":1}
I20260808 08:22:29.119035 31065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:29.119336 31065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:29.119410 31065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55cbf1d9b94048fd9cc6e4373c879aab, State: Initialized, Role: FOLLOWER
I20260808 08:22:29.119904 31065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:29.120023 31065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:22:29.120064 31065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:22:29.120131 31065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:29.121275 31065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:29.121644 31065 leader_election.cc:302] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55cbf1d9b94048fd9cc6e4373c879aab; no voters:
I20260808 08:22:29.122052 31065 leader_election.cc:288] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:22:29.122155 31075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:29.122478 31075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 LEADER]: Becoming Leader. State: Replica: 55cbf1d9b94048fd9cc6e4373c879aab, State: Running, Role: LEADER
I20260808 08:22:29.122877 31075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:29.123559 31065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:29.124883 31078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } } }
I20260808 08:22:29.125008 31078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:29.125229 31078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55cbf1d9b94048fd9cc6e4373c879aab. Latest consensus state: current_term: 1 leader_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } } }
I20260808 08:22:29.125295 31078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:29.125705 31087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:29.126171 31065 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 08:22:29.132913 31087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:29.139940 31087 catalog_manager.cc:1347] Generated new cluster ID: e55ee426c668437b98572c4b4725f70e
I20260808 08:22:29.140030 31087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:22:29.155169 31087 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:22:29.156582 31087 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:22:29.162031 31087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Generated new TSK 0
I20260808 08:22:29.163002 31087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:22:29.895373 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46183
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:30.033245 31137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:30.033569 31137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:30.033670 31137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:30.037941 31137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:30.038046 31137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:30.038141 31137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:22:30.043835 31137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.13.20:50517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:30.045055 31137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:30.046559 31137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:30.058493 31146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:30.058507 31149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:30.058900 31145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:30.060053 31137 server_base.cc:1034] running on GCE node
I20260808 08:22:30.060563 31137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:30.061384 31137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:30.062580 31137 hybrid_clock.cc:648] HybridClock initialized: now 1786177350062548 us; error 47 us; skew 500 ppm
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:30.066612 31137 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:22:30.067842 31137 webserver.cc:460] Webserver started at http://127.12.13.1:35529/ using document root <none> and password file <none>
I20260808 08:22:30.068691 31137 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:30.068849 31137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:30.069157 31137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:30.088567 31137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "c6899cd724704b5292364923eaf6af59"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "76e5d55fb60bd93e902f7ba5a450e6a3"
server_key_iv: "f52f9dbc93106ea032af72f91e218b0c"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.089470 31137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "c6899cd724704b5292364923eaf6af59"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "76e5d55fb60bd93e902f7ba5a450e6a3"
server_key_iv: "f52f9dbc93106ea032af72f91e218b0c"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.095475 31137 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 08:22:30.115717 31159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.117316 31137 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 08:22:30.117486 31137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "c6899cd724704b5292364923eaf6af59"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "76e5d55fb60bd93e902f7ba5a450e6a3"
server_key_iv: "f52f9dbc93106ea032af72f91e218b0c"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.117645 31137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:30.149358 31137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:30.153407 31137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:30.153700 31137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:30.154551 31137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:30.155922 31137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:30.156055 31137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 08:22:30.156204 31137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:30.156294 31137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.178496 31137 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:45881
I20260808 08:22:30.179924 31137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:22:30.185701 31334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:45881 every 8 connection(s)
I20260808 08:22:30.198237 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 31137
I20260808 08:22:30.198411 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:22:30.198752 12340 external_mini_cluster.cc:1428] Setting key 5ccfff759c21f314ba05518f8e7acc89
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:30.212607 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:30.184509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:55929 (local address 127.12.13.62:46183)
0808 08:22:30.188314 (+ 3805us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:30.188321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:30.188338 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:22:30.188395 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:30.188400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:30.188463 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:30.188568 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:30.199982 (+ 11414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.200760 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.202783 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.203200 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.203298 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:30.207041 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:30.207048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:30.207054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:30.207096 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:30.209348 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:30.210158 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:30.210162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:30.210164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:30.210271 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:30.210757 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:30.210763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:30.210764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:30.211041 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:30.211140 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:30.212165 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0808 08:22:30.212348 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3700,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:22:30.216393 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:30.183238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46183 (local address 127.12.13.1:55929)
0808 08:22:30.186429 (+ 3191us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:30.186493 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:30.187800 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:30.189058 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:30.189071 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:30.189479 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:30.199725 (+ 10246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:30.199786 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.200973 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.200977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:30.202612 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:30.202630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.203602 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.203608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:30.203880 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:30.204702 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:30.204728 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:30.206795 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:30.209512 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:30.209522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:30.209548 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:30.210013 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:30.210413 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:30.210418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:30.210421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:30.210633 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:30.211221 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:30.211231 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:30.211760 (+ 529us) client_negotiation.cc:769] Sending connection context
0808 08:22:30.214728 (+ 2968us) client_negotiation.cc:240] Negotiation successful
0808 08:22:30.215143 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2919,"thread_start_us":160,"threads_started":1}
I20260808 08:22:30.218743 31335 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46183
I20260808 08:22:30.219054 31335 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:30.219655 31335 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:22:30.221822 31010 ts_manager.cc:194] Registered new tserver with Master: c6899cd724704b5292364923eaf6af59 (127.12.13.1:45881)
I20260808 08:22:30.223488 31010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:55929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:22:30.250809 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46183
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 08:22:30.425987 31349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:30.426407 31349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:30.426483 31349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:30.432592 31349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:30.432680 31349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:30.432776 31349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:22:30.440833 31349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.13.20:50517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:30.442440 31349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:30.443967 31349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:30.457966 31358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:30.458119 31362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:30.458026 31357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:30.459275 31349 server_base.cc:1034] running on GCE node
I20260808 08:22:30.459875 31349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:30.460708 31349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:30.468163 31349 hybrid_clock.cc:648] HybridClock initialized: now 1786177350468074 us; error 101 us; skew 500 ppm
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:30.472298 31349 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:22:30.473702 31349 webserver.cc:460] Webserver started at http://127.12.13.2:46787/ using document root <none> and password file <none>
I20260808 08:22:30.474601 31349 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:30.474712 31349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:30.475031 31349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:30.497972 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "37b3bbc7272b4f6983250e43f4ad38fb"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "56c4f7533f50f1bf60a8893f0fcce577"
server_key_iv: "e3c32641f16a1ccde220a8d06c393546"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.498968 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "37b3bbc7272b4f6983250e43f4ad38fb"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "56c4f7533f50f1bf60a8893f0fcce577"
server_key_iv: "e3c32641f16a1ccde220a8d06c393546"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.505177 31349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 08:22:30.531625 31374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.533648 31349 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260808 08:22:30.533811 31349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "37b3bbc7272b4f6983250e43f4ad38fb"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "56c4f7533f50f1bf60a8893f0fcce577"
server_key_iv: "e3c32641f16a1ccde220a8d06c393546"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.533993 31349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:30.552461 31349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:30.556557 31349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:30.556847 31349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:30.557623 31349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:30.558763 31349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:30.558845 31349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.558925 31349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:30.558962 31349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.571801 31349 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:42543
I20260808 08:22:30.573001 31349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:22:30.574319 31565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:42543 every 8 connection(s)
I20260808 08:22:30.583071 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 31349
I20260808 08:22:30.583217 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:22:30.583573 12340 external_mini_cluster.cc:1428] Setting key 7ceedd79157adb954a82a31525e6cf5d
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:30.589239 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:30.575478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:60723 (local address 127.12.13.62:46183)
0808 08:22:30.575625 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:30.575633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:30.576990 (+ 1357us) server_negotiation.cc:407] Connection header received
0808 08:22:30.578235 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:30.578239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:30.578299 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:30.578379 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:30.579527 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.580080 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.581249 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.581498 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.581617 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:30.584756 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:30.584760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:30.584762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:30.584793 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:30.586677 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:30.587335 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:30.587338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:30.587340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:30.587442 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:30.587861 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:30.587864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:30.587866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:30.588092 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:30.588166 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:30.588919 (+ 753us) server_negotiation.cc:299] Negotiation successful
0808 08:22:30.589059 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:22:30.591039 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:30.576093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46183 (local address 127.12.13.2:60723)
0808 08:22:30.576824 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:30.576883 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:30.578007 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:30.578564 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:30.578571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:30.578853 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:30.579374 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:30.579388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.580226 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.580230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:30.581118 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:30.581128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.581662 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.581666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:30.581856 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:30.582782 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:30.582804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:30.584565 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:30.586830 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:30.586837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:30.586855 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:30.587209 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:30.587560 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:30.587563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:30.587565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:30.587748 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:30.588253 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:30.588260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:30.588625 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 08:22:30.589572 (+ 947us) client_negotiation.cc:240] Negotiation successful
0808 08:22:30.589952 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":167,"threads_started":1}
I20260808 08:22:30.592464 31567 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46183
I20260808 08:22:30.592743 31567 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:30.593532 31567 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:30.594712 31010 ts_manager.cc:194] Registered new tserver with Master: 37b3bbc7272b4f6983250e43f4ad38fb (127.12.13.2:42543)
I20260808 08:22:30.595450 31010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:60723
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:22:30.650795 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46183
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
W20260808 08:22:30.826613 31575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:30.827008 31575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:30.827069 31575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:30.833011 31575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:30.833101 31575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:30.833185 31575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:22:30.840869 31575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.13.20:50517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:30.842247 31575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:30.843407 31575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:30.854120 31592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:30.854099 31585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:30.854218 31575 server_base.cc:1034] running on GCE node
W20260808 08:22:30.854362 31589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:30.854840 31575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:30.855615 31575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:30.858912 31575 hybrid_clock.cc:648] HybridClock initialized: now 1786177350858894 us; error 38 us; skew 500 ppm
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:30.867321 31575 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:22:30.868669 31575 webserver.cc:460] Webserver started at http://127.12.13.3:39615/ using document root <none> and password file <none>
I20260808 08:22:30.869493 31575 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:30.869597 31575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:30.869917 31575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:30.890463 31575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "4ab91b9963844d86a17605c145dec508"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "50b1b883826e00c51f7fbda96cedee51"
server_key_iv: "f380151949c71c803a69051c40d9ead6"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.892285 31575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "4ab91b9963844d86a17605c145dec508"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "50b1b883826e00c51f7fbda96cedee51"
server_key_iv: "f380151949c71c803a69051c40d9ead6"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.899436 31575 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260808 08:22:30.929178 31600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.003s user 0.000s sys 0.000s
I20260808 08:22:30.930985 31575 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260808 08:22:30.931160 31575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "4ab91b9963844d86a17605c145dec508"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "50b1b883826e00c51f7fbda96cedee51"
server_key_iv: "f380151949c71c803a69051c40d9ead6"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:30.931329 31575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:30.948464 31575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:30.952462 31575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:30.952736 31575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:30.953548 31575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:30.954751 31575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:30.954839 31575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.954941 31575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:30.954999 31575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:30.968119 31575 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:35197
I20260808 08:22:30.969571 31575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:22:30.971673 31809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:35197 every 8 connection(s)
I20260808 08:22:30.975533 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 31575
I20260808 08:22:30.975678 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:22:30.976019 12340 external_mini_cluster.cc:1428] Setting key 7a9b92a9a8442aef3555978346c7c47b
Aug 08 08:22:30 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:30.995819 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:30.973764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:55685 (local address 127.12.13.62:46183)
0808 08:22:30.974166 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:30.974170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:30.975068 (+ 898us) server_negotiation.cc:407] Connection header received
0808 08:22:30.976528 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:30.976532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:30.976592 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:30.976671 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:30.979283 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.980132 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.982020 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.982390 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.982489 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:30.990375 (+ 7886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:30.990380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:30.990383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:30.990419 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:30.992464 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:30.993324 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:30.993328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:30.993330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:30.993433 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:30.993951 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:30.993954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:30.993955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:30.994227 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:30.994343 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:30.995450 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0808 08:22:30.995613 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260808 08:22:30.998106 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:30.974173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46183 (local address 127.12.13.3:55685)
0808 08:22:30.974876 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:30.974945 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:30.976236 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:30.977210 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:30.977279 (+ 69us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:30.977676 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:30.979074 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:30.979098 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.980332 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.980337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:30.981836 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:30.981849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:30.982731 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:30.982735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:30.982957 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:30.983778 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:30.983800 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:30.990097 (+ 6297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:30.992649 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:30.992657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:30.992681 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:30.993170 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:30.993565 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:30.993568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:30.993570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:30.993804 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:30.994427 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:30.994435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:30.995075 (+ 640us) client_negotiation.cc:769] Sending connection context
0808 08:22:30.996363 (+ 1288us) client_negotiation.cc:240] Negotiation successful
0808 08:22:30.996831 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":210,"threads_started":1}
I20260808 08:22:31.000026 31813 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46183
I20260808 08:22:31.000384 31813 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:31.001080 31813 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:31.003746 31010 ts_manager.cc:194] Registered new tserver with Master: 4ab91b9963844d86a17605c145dec508 (127.12.13.3:35197)
I20260808 08:22:31.004900 31010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:55685
I20260808 08:22:31.021337 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:22:31.022312 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 31137
I20260808 08:22:31.067831 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 31349
I20260808 08:22:31.085631 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 31575
I20260808 08:22:31.117105 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 30967
I20260808 08:22:31.132081 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46183
--webserver_interface=127.12.13.62
--webserver_port=32851
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:46183
--ranger_config_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:31.319131 31821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:31.319574 31821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:31.319705 31821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:31.327523 31821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:31.327623 31821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:31.327656 31821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:31.327678 31821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:31.327699 31821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:31.334614 31821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:46183
--ranger_config_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:31.335848 31821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:31.336970 31821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:31.346654 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:31.346791 31821 server_base.cc:1034] running on GCE node
W20260808 08:22:31.346609 31835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:31.346635 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:31.347618 31821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:31.348878 31821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:31.350080 31821 hybrid_clock.cc:648] HybridClock initialized: now 1786177351350066 us; error 28 us; skew 500 ppm
Aug 08 08:22:31 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:31.353798 31821 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:31.354964 31821 webserver.cc:460] Webserver started at http://127.12.13.62:32851/ using document root <none> and password file <none>
I20260808 08:22:31.355623 31821 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:31.355744 31821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 08:22:31 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:31.407016 31821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 08:22:31.410295 31847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:31.412076 31821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 08:22:31.412230 31821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "55cbf1d9b94048fd9cc6e4373c879aab"
format_stamp: "Formatted at 2026-08-08 08:22:29 on dist-test-slave-6ntk"
server_key: "21b5900bead81b558a743d9b56aaa267"
server_key_iv: "fda9b3e23ece8d451e1c76460c2d7a29"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:31.413017 31821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:31.437985 31821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:31.441543 31821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:31.441718 31821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:31.453702 31821 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:46183
I20260808 08:22:31.453825 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:46183 every 8 connection(s)
I20260808 08:22:31.455149 31821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:31.460738 31934 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:22:31.462764 31934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Bootstrap starting.
I20260808 08:22:31.462790 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 31821
I20260808 08:22:31.463393 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:45881
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:46183
--webserver_port=35529
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:31.513051 31934 log.cc:824] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:31.526376 31934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:22:31.526909 31934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Bootstrap complete.
I20260808 08:22:31.531556 31934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:31.531981 31934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55cbf1d9b94048fd9cc6e4373c879aab, State: Initialized, Role: FOLLOWER
I20260808 08:22:31.532541 31934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:31.532696 31934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:22:31.532783 31934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:22:31.532917 31934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 1 FOLLOWER]: Advancing to term 2
I20260808 08:22:31.534970 31934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:31.535326 31934 leader_election.cc:302] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55cbf1d9b94048fd9cc6e4373c879aab; no voters:
I20260808 08:22:31.535728 31934 leader_election.cc:288] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 08:22:31.535827 31938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 2 FOLLOWER]: Leader election won for term 2
I20260808 08:22:31.536185 31938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [term 2 LEADER]: Becoming Leader. State: Replica: 55cbf1d9b94048fd9cc6e4373c879aab, State: Running, Role: LEADER
I20260808 08:22:31.536540 31938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } }
I20260808 08:22:31.537147 31934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:31.537734 31938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } } }
I20260808 08:22:31.537967 31938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:31.538240 31938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55cbf1d9b94048fd9cc6e4373c879aab. Latest consensus state: current_term: 2 leader_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cbf1d9b94048fd9cc6e4373c879aab" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46183 } } }
I20260808 08:22:31.538398 31938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:31.538796 31944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:31.542631 31944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:31.543246 31934 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 08:22:31.546108 31944 catalog_manager.cc:1259] Loaded cluster ID: e55ee426c668437b98572c4b4725f70e
I20260808 08:22:31.546226 31944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:22:31.548669 31944 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:22:31.549412 31944 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Loaded TSK: 0
I20260808 08:22:31.561093 31944 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 08:22:31.656838 31936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:31.657317 31936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:31.657442 31936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:31.663895 31936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:31.664057 31936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:31.664227 31936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:22:31.672681 31936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:45881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=35529
--tserver_master_addrs=127.12.13.62:46183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:31.674548 31936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:31.676077 31936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:31.692276 31992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:31.692198 31990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:31.694425 31936 server_base.cc:1034] running on GCE node
W20260808 08:22:31.698551 31989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:31.699486 31936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:31.700464 31936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:31.703779 31936 hybrid_clock.cc:648] HybridClock initialized: now 1786177351703722 us; error 65 us; skew 500 ppm
Aug 08 08:22:31 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:31.709110 31936 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:22:31.710405 31936 webserver.cc:460] Webserver started at http://127.12.13.1:35529/ using document root <none> and password file <none>
I20260808 08:22:31.712453 31936 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:31.712634 31936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 08:22:31 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:31.780017 31936 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.004s sys 0.000s
I20260808 08:22:31.788255 32007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:31.790692 31936 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260808 08:22:31.790854 31936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "c6899cd724704b5292364923eaf6af59"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "76e5d55fb60bd93e902f7ba5a450e6a3"
server_key_iv: "f52f9dbc93106ea032af72f91e218b0c"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:31.791677 31936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:31.826735 31936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:31.831125 31936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:31.831422 31936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:31.832264 31936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:31.833328 31936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:31.833451 31936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:31.833602 31936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:31.833719 31936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:31.849653 31936 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:45881
I20260808 08:22:31.849789 32185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:45881 every 8 connection(s)
I20260808 08:22:31.851330 31936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:22:31.855095 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 31936
I20260808 08:22:31.855681 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:42543
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:46183
--webserver_port=46787
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:22:31 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:31.876492 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:31.854171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59161 (local address 127.12.13.62:46183)
0808 08:22:31.855405 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:31.855436 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:31.857926 (+ 2490us) server_negotiation.cc:407] Connection header received
0808 08:22:31.859430 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:31.859448 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:31.859915 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:31.860314 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:31.861822 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:31.863489 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:31.865292 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:31.866083 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:31.866281 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:31.869630 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:31.869639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:31.869661 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:31.869702 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:31.872057 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:31.872840 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:31.872848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:31.872855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:31.873027 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:31.873512 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:31.873518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:31.873520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:31.874116 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:31.874246 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:31.875147 (+ 901us) server_negotiation.cc:299] Negotiation successful
0808 08:22:31.875532 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":132,"threads_started":1}
I20260808 08:22:31.878090 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:31.854262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46183 (local address 127.12.13.1:59161)
0808 08:22:31.857652 (+ 3390us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:31.857724 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:31.858865 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:31.860460 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:31.860470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:31.860864 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:31.861596 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:31.861621 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:31.863669 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:31.863674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:31.865134 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:31.865153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:31.866583 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:31.866587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:31.866801 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:31.867583 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:31.867609 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:31.869309 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:31.872225 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:31.872232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:31.872249 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:31.872673 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:31.873162 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:31.873165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:31.873167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:31.873385 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:31.874341 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:31.874359 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:31.874726 (+ 367us) client_negotiation.cc:769] Sending connection context
0808 08:22:31.876906 (+ 2180us) client_negotiation.cc:240] Negotiation successful
0808 08:22:31.877270 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3122,"thread_start_us":163,"threads_started":1}
I20260808 08:22:31.880533 32186 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46183
I20260808 08:22:31.880857 32186 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:31.881572 32186 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
W20260808 08:22:31.884384 32186 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:46183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 08:22:31.884534 32186 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:31.884732 32186 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:31.885767 32186 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:31.886003 32186 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
W20260808 08:22:31.886814 32186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:22:32.024351 32192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:32.024647 32192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:32.024729 32192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:32.029501 32192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:32.029599 32192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:32.029706 32192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:22:32.035496 32192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:42543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=46787
--tserver_master_addrs=127.12.13.62:46183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:32.036772 32192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:32.038015 32192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:32.050815 32200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:32.050815 32199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:32.052389 32202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:32.052904 32192 server_base.cc:1034] running on GCE node
I20260808 08:22:32.053387 32192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:32.054246 32192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:32.055457 32192 hybrid_clock.cc:648] HybridClock initialized: now 1786177352055428 us; error 39 us; skew 500 ppm
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:32.058861 32192 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:22:32.059945 32192 webserver.cc:460] Webserver started at http://127.12.13.2:46787/ using document root <none> and password file <none>
I20260808 08:22:32.060638 32192 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:32.060755 32192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:32.086122 32192 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 08:22:32.089861 32214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:32.091682 32192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 08:22:32.091848 32192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "37b3bbc7272b4f6983250e43f4ad38fb"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "56c4f7533f50f1bf60a8893f0fcce577"
server_key_iv: "e3c32641f16a1ccde220a8d06c393546"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:32.092676 32192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:32.112229 32192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:32.117213 32192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:32.117537 32192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:32.118616 32192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:32.120047 32192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:32.120177 32192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:32.120307 32192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:32.120388 32192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:32.132431 32192 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:42543
I20260808 08:22:32.132571 32385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:42543 every 8 connection(s)
I20260808 08:22:32.134029 32192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:22:32.138949 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 32192
I20260808 08:22:32.139511 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.13.20:50517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:35197
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:46183
--webserver_port=39615
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:32.155722 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:32.138039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:55823 (local address 127.12.13.62:46183)
0808 08:22:32.138261 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:32.138267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:32.140352 (+ 2085us) server_negotiation.cc:407] Connection header received
0808 08:22:32.141701 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:32.141706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:32.141803 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:32.141906 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:32.143817 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.144656 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.146655 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.147005 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.147139 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:32.150667 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:32.150671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:32.150674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:32.150712 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:32.152716 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:32.153491 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:32.153496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:32.153498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:32.153599 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:32.154131 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:32.154137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:32.154140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:32.154401 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:32.154492 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:32.155329 (+ 837us) server_negotiation.cc:299] Negotiation successful
0808 08:22:32.155499 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 08:22:32.158012 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:32.138366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46183 (local address 127.12.13.2:55823)
0808 08:22:32.140151 (+ 1785us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:32.140223 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:32.141418 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:32.142376 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:32.142386 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:32.142776 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:32.143617 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:32.143641 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.144905 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.144909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:32.146497 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:32.146510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.147349 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.147354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:32.147579 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:32.148345 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:32.148370 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:32.150403 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:32.152887 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:32.152894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:32.152916 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:32.153342 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:32.153724 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:32.153727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:32.153729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:32.154006 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:32.154568 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:32.154576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:32.155006 (+ 430us) client_negotiation.cc:769] Sending connection context
0808 08:22:32.156272 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0808 08:22:32.156680 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1538,"thread_start_us":149,"threads_started":1}
I20260808 08:22:32.159802 32388 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46183
I20260808 08:22:32.160069 32388 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.160689 32388 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
W20260808 08:22:32.162619 32388 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:46183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 08:22:32.162744 32388 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.162936 32388 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:32.163839 32388 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.164004 32388 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
W20260808 08:22:32.164799 32388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:22:32.322598 32394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:32.322921 32394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:32.322995 32394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:32.329023 32394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:32.329171 32394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:32.329289 32394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:22:32.337436 32394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.13.20:50517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:35197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=39615
--tserver_master_addrs=127.12.13.62:46183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:32.339475 32394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:32.341321 32394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:32.356109 32408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:32.357815 32406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:32.358419 32394 server_base.cc:1034] running on GCE node
W20260808 08:22:32.358769 32402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:32.359407 32394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:32.360366 32394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:32.364658 32394 hybrid_clock.cc:648] HybridClock initialized: now 1786177352364564 us; error 100 us; skew 500 ppm
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:32.369264 32394 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:22:32.370697 32394 webserver.cc:460] Webserver started at http://127.12.13.3:39615/ using document root <none> and password file <none>
I20260808 08:22:32.371647 32394 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:32.371855 32394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:32.396554 32394 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 08:22:32.401026 32416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:32.403188 32394 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260808 08:22:32.403369 32394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "4ab91b9963844d86a17605c145dec508"
format_stamp: "Formatted at 2026-08-08 08:22:30 on dist-test-slave-6ntk"
server_key: "50b1b883826e00c51f7fbda96cedee51"
server_key_iv: "f380151949c71c803a69051c40d9ead6"
server_key_version: "kuduclusterkey@0"
I20260808 08:22:32.404229 32394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:32.431707 32394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:32.435717 32394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:32.436041 32394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:32.436933 32394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:32.438321 32394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:32.438556 32394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:32.438696 32394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:32.438781 32394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:32.452327 32394 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:35197
I20260808 08:22:32.458211 32619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:35197 every 8 connection(s)
I20260808 08:22:32.470198 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 32394
I20260808 08:22:32.472270 32394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:22:32.484474 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:32.471046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.12.13.62:46183)
0808 08:22:32.471247 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:32.471253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:32.472647 (+ 1394us) server_negotiation.cc:407] Connection header received
0808 08:22:32.472838 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:32.472842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:32.472906 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:32.473003 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:32.474511 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.475437 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.477084 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.477468 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.477592 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:32.480402 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:32.480407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:32.480410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:32.480445 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:32.482343 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:32.482954 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:32.482958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:32.482960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:32.483070 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:32.483391 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:32.483394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:32.483396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:32.483649 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:32.483932 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:32.484163 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 08:22:32.484304 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:32.502836 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:32.478127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:56471 (local address 127.12.13.62:46183)
0808 08:22:32.480545 (+ 2418us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:32.480550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:32.482984 (+ 2434us) server_negotiation.cc:407] Connection header received
0808 08:22:32.485237 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:32.485241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:32.485301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:32.485391 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:32.487719 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.488640 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.490772 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.491192 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.491292 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:32.494913 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:32.494918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:32.494921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:32.494957 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:32.497184 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:32.500234 (+ 3050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:32.500241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:32.500243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:32.500388 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:32.500991 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:32.500995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:32.500996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:32.501305 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:32.501415 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:32.502411 (+ 996us) server_negotiation.cc:299] Negotiation successful
0808 08:22:32.502593 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2329,"thread_start_us":113,"threads_started":1}
I20260808 08:22:32.505208 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:32.478691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46183 (local address 127.12.13.3:56471)
0808 08:22:32.481211 (+ 2520us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:32.481286 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:32.484777 (+ 3491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:32.486084 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:32.486096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:32.486657 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:32.487491 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:32.487518 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.488879 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.488883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:32.490580 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:32.490594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:32.491548 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:32.491553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:32.491821 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:32.492647 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:32.492685 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:32.494639 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:32.497362 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:32.497370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:32.497392 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:32.497864 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:32.500546 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:32.500549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:32.500551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:32.500846 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:32.501506 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:32.501515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:32.502037 (+ 522us) client_negotiation.cc:769] Sending connection context
0808 08:22:32.503435 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0808 08:22:32.503892 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1919,"thread_start_us":168,"threads_started":1}
I20260808 08:22:32.511067 32624 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46183
I20260808 08:22:32.511407 32624 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.512231 32624 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
W20260808 08:22:32.514501 32624 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:46183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 08:22:32.514680 32624 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.514906 32624 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:32.516034 32624 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.516250 32624 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
W20260808 08:22:32.517371 32624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 08:22:32.887049 32186 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:32.887428 32186 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:32.888659 31873 ts_manager.cc:194] Registered new tserver with Master: c6899cd724704b5292364923eaf6af59 (127.12.13.1:45881)
I20260808 08:22:32.890036 31873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:59161
Aug 08 08:22:32 dist-test-slave-6ntk krb5kdc[30268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:33.165056 32388 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:33.165412 32388 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:33.166438 31873 ts_manager.cc:194] Registered new tserver with Master: 37b3bbc7272b4f6983250e43f4ad38fb (127.12.13.2:42543)
I20260808 08:22:33.167264 31873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:55823
Aug 08 08:22:33 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
I20260808 08:22:33.517987 32624 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:33.518355 32624 heartbeater.cc:511] Master 127.12.13.62:46183 requested a full tablet report, sending...
I20260808 08:22:33.519322 31873 ts_manager.cc:194] Registered new tserver with Master: 4ab91b9963844d86a17605c145dec508 (127.12.13.3:35197)
I20260808 08:22:33.520227 31873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:56471
I20260808 08:22:33.523485 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:22:33.540184 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.525558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.12.13.62:46183)
0808 08:22:33.528404 (+ 2846us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.528412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.528432 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:22:33.528488 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.528492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.528554 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.528658 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:33.529596 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.530499 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.531655 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.532018 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.532152 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.535170 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:33.535174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:33.535178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:33.535212 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:33.537411 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:33.537975 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:33.537979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:33.537981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:33.538093 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:33.538543 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:33.538547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:33.538549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:33.538839 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:33.539769 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.539852 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.539990 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2711,"thread_start_us":96,"threads_started":1}
I20260808 08:22:33.551460 31872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Aug 08 08:22:33 dist-test-slave-6ntk krb5kdc[30268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for HTTP/127.12.13.20@KRBTEST.COM
W20260808 08:22:33.556923 31872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:22:33.592374 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.572172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:45881 (local address 127.0.0.1:45936)
0808 08:22:33.577570 (+ 5398us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.577635 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:33.577773 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:33.579078 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:33.579085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:33.579111 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:33.579253 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.579260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.585286 (+ 6026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.585292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.586790 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.586802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.591728 (+ 4926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.591732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.591885 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.591930 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 08:22:33.592052 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 08:22:33.592197 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5185,"thread_start_us":115,"threads_started":1}
I20260808 08:22:33.593580 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.576032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.12.13.2:42543)
0808 08:22:33.578090 (+ 2058us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.578099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.580767 (+ 2668us) server_negotiation.cc:407] Connection header received
0808 08:22:33.580871 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.580877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.581085 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.581225 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:33.582394 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.583530 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.589209 (+ 5679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.590154 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.590261 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.590428 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.593307 (+ 2879us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.593432 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1940,"thread_start_us":100,"threads_started":1}
I20260808 08:22:33.593194 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.575605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:42543 (local address 127.0.0.1:48598)
0808 08:22:33.580555 (+ 4950us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.580568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:33.580662 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:33.582033 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:33.582037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:33.582144 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:33.582250 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.582258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.587018 (+ 4760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.587021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.588160 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.588173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.592754 (+ 4581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.592758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.592866 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.592892 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 08:22:33.592964 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 08:22:33.593037 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4886,"thread_start_us":80,"threads_started":1}
I20260808 08:22:33.595429 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.573208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.12.13.1:45881)
0808 08:22:33.578292 (+ 5084us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.578299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.578324 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:22:33.578410 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.578415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.578647 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.578782 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:33.581398 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.585151 (+ 3753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.590504 (+ 5353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.591400 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.591512 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.591656 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.595161 (+ 3505us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.595290 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4976,"thread_start_us":78,"threads_started":1}
I20260808 08:22:33.599635 32289 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d (DEFAULT_TABLE table=test-table [id=4038d6e519d74db292912260b3b625cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:33.602663 32061 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d (DEFAULT_TABLE table=test-table [id=4038d6e519d74db292912260b3b625cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:33.604090 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe7a490e9b4449953ea8802d3e730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:33.606215 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.575266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:35197 (local address 127.0.0.1:51808)
0808 08:22:33.579469 (+ 4203us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.579481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:33.579554 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:33.585104 (+ 5550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:33.585107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:33.585126 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:33.585250 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.585259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.593139 (+ 7880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.593141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.594238 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.594250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.596147 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.596153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.596264 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.596288 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:22:33.596379 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 08:22:33.606037 (+ 9658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4139,"thread_start_us":116,"threads_started":1}
I20260808 08:22:33.607029 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.575447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.12.13.3:35197)
0808 08:22:33.584495 (+ 9048us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.584503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.584528 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:22:33.584601 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.584606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.584805 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.584950 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:33.586355 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.593007 (+ 6652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.594966 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.595824 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.595933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.596077 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.606726 (+ 10649us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.606875 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8921,"thread_start_us":73,"threads_started":1}
I20260808 08:22:33.607612 32289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe7a490e9b4449953ea8802d3e730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:33.610668 32482 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d (DEFAULT_TABLE table=test-table [id=4038d6e519d74db292912260b3b625cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:33.611896 32482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe7a490e9b4449953ea8802d3e730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:33.623845 32671 tablet_bootstrap.cc:492] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: Bootstrap starting.
I20260808 08:22:33.626669 32671 tablet_bootstrap.cc:654] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:33.627766 32671 log.cc:824] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:33.628851 32675 tablet_bootstrap.cc:492] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Bootstrap starting.
I20260808 08:22:33.629355 32678 tablet_bootstrap.cc:492] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59: Bootstrap starting.
I20260808 08:22:33.632068 32675 tablet_bootstrap.cc:654] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:33.635931 32678 tablet_bootstrap.cc:654] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:33.637027 32675 log.cc:824] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:33.638103 32678 log.cc:824] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:33.642102 32671 tablet_bootstrap.cc:492] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: No bootstrap required, opened a new log
I20260808 08:22:33.642426 32671 ts_tablet_manager.cc:1397] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20260808 08:22:33.648290 32671 raft_consensus.cc:348] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.648815 32671 raft_consensus.cc:374] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:33.648934 32671 raft_consensus.cc:729] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b3bbc7272b4f6983250e43f4ad38fb, State: Initialized, Role: FOLLOWER
I20260808 08:22:33.649603 32671 consensus_queue.cc:260] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.651074 32671 ts_tablet_manager.cc:1428] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260808 08:22:33.651535 32388 heartbeater.cc:503] Master 127.12.13.62:46183 was elected leader, sending a full tablet report...
I20260808 08:22:33.667707 32678 tablet_bootstrap.cc:492] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59: No bootstrap required, opened a new log
I20260808 08:22:33.667707 32675 tablet_bootstrap.cc:492] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: No bootstrap required, opened a new log
I20260808 08:22:33.667991 32678 ts_tablet_manager.cc:1397] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.002s
I20260808 08:22:33.667991 32675 ts_tablet_manager.cc:1397] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20260808 08:22:33.672407 32678 raft_consensus.cc:348] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.672407 32675 raft_consensus.cc:348] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.672732 32678 raft_consensus.cc:374] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:33.672732 32675 raft_consensus.cc:374] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:33.672797 32675 raft_consensus.cc:729] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab91b9963844d86a17605c145dec508, State: Initialized, Role: FOLLOWER
I20260808 08:22:33.672791 32678 raft_consensus.cc:729] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6899cd724704b5292364923eaf6af59, State: Initialized, Role: FOLLOWER
I20260808 08:22:33.673573 32678 consensus_queue.cc:260] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.674971 32678 ts_tablet_manager.cc:1428] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 08:22:33.675532 32186 heartbeater.cc:503] Master 127.12.13.62:46183 was elected leader, sending a full tablet report...
I20260808 08:22:33.677654 32675 consensus_queue.cc:260] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.678915 32675 ts_tablet_manager.cc:1428] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260808 08:22:33.679054 32624 heartbeater.cc:503] Master 127.12.13.62:46183 was elected leader, sending a full tablet report...
I20260808 08:22:33.703804 32683 raft_consensus.cc:482] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:33.704097 32683 raft_consensus.cc:504] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.705757 32683 leader_election.cc:288] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab91b9963844d86a17605c145dec508 (127.12.13.3:35197), 37b3bbc7272b4f6983250e43f4ad38fb (127.12.13.2:42543)
I20260808 08:22:33.716305 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.707061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40419 (local address 127.12.13.3:35197)
0808 08:22:33.707234 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.707240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.711478 (+ 4238us) server_negotiation.cc:407] Connection header received
0808 08:22:33.711724 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.711727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.711789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.711870 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:33.712357 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.713138 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.714880 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.715494 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.715582 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.715661 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.716044 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.716132 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:22:33.717053 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.706066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:35197 (local address 127.12.13.1:40419)
0808 08:22:33.711353 (+ 5287us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.711377 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:33.711585 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:33.712003 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:33.712006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:33.712026 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:33.712182 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.712190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.713316 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.713319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.714719 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.714732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.715772 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.715776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.715880 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.715908 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 08:22:33.716796 (+ 888us) client_negotiation.cc:240] Negotiation successful
0808 08:22:33.716892 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5128,"thread_start_us":149,"threads_started":1}
I20260808 08:22:33.718066 32539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe7a490e9b4449953ea8802d3e730d" candidate_uuid: "c6899cd724704b5292364923eaf6af59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab91b9963844d86a17605c145dec508" is_pre_election: true
I20260808 08:22:33.718490 32539 raft_consensus.cc:2454] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6899cd724704b5292364923eaf6af59 in term 0.
I20260808 08:22:33.719153 32014 leader_election.cc:302] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab91b9963844d86a17605c145dec508, c6899cd724704b5292364923eaf6af59; no voters:
I20260808 08:22:33.719527 32683 raft_consensus.cc:2781] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:22:33.719645 32683 raft_consensus.cc:482] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:22:33.719700 32683 raft_consensus.cc:3037] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:33.721207 32683 raft_consensus.cc:504] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
I20260808 08:22:33.721658 32683 leader_election.cc:288] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [CANDIDATE]: Term 1 election: Requested vote from peers 4ab91b9963844d86a17605c145dec508 (127.12.13.3:35197), 37b3bbc7272b4f6983250e43f4ad38fb (127.12.13.2:42543)
I20260808 08:22:33.722458 32539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe7a490e9b4449953ea8802d3e730d" candidate_uuid: "c6899cd724704b5292364923eaf6af59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab91b9963844d86a17605c145dec508"
I20260808 08:22:33.722594 32539 raft_consensus.cc:3037] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:33.724023 32539 raft_consensus.cc:2454] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6899cd724704b5292364923eaf6af59 in term 1.
I20260808 08:22:33.724493 32014 leader_election.cc:302] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab91b9963844d86a17605c145dec508, c6899cd724704b5292364923eaf6af59; no voters:
I20260808 08:22:33.724741 32683 raft_consensus.cc:2781] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:33.724977 32683 raft_consensus.cc:686] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [term 1 LEADER]: Becoming Leader. State: Replica: c6899cd724704b5292364923eaf6af59, State: Running, Role: LEADER
I20260808 08:22:33.725405 32683 consensus_queue.cc:237] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } }
W20260808 08:22:33.729002 32625 tablet.cc:2369] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:22:33.729419 31873 catalog_manager.cc:5495] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6899cd724704b5292364923eaf6af59 (127.12.13.1). New cstate: current_term: 1 leader_uuid: "c6899cd724704b5292364923eaf6af59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6899cd724704b5292364923eaf6af59" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 } health_report { overall_health: UNKNOWN } } }
I20260808 08:22:33.735069 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.707211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:37699 (local address 127.12.13.2:42543)
0808 08:22:33.707337 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.707341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.717912 (+ 10571us) server_negotiation.cc:407] Connection header received
0808 08:22:33.718756 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.718760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.718827 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.718882 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:33.731027 (+ 12145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.731848 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.733972 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.734411 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.734570 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.734643 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.734835 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.734931 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 08:22:33.735754 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.706551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:42543 (local address 127.12.13.1:37699)
0808 08:22:33.717775 (+ 11224us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.717790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:33.718029 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:33.719097 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:33.719100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:33.719121 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:33.719202 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.719207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.732705 (+ 13498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.732709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.733802 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:33.733817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.734570 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.734574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:33.734664 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.734698 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 08:22:33.735498 (+ 800us) client_negotiation.cc:240] Negotiation successful
0808 08:22:33.735598 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11158,"thread_start_us":101,"threads_started":1}
I20260808 08:22:33.743043 32317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe7a490e9b4449953ea8802d3e730d" candidate_uuid: "c6899cd724704b5292364923eaf6af59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" is_pre_election: true
I20260808 08:22:33.743448 32317 raft_consensus.cc:2454] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6899cd724704b5292364923eaf6af59 in term 0.
I20260808 08:22:33.743845 32316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe7a490e9b4449953ea8802d3e730d" candidate_uuid: "c6899cd724704b5292364923eaf6af59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b3bbc7272b4f6983250e43f4ad38fb"
I20260808 08:22:33.743963 32316 raft_consensus.cc:3037] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:33.745385 32316 raft_consensus.cc:2454] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6899cd724704b5292364923eaf6af59 in term 1.
I20260808 08:22:33.753028 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.747744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.12.13.1:45881)
0808 08:22:33.747874 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:33.747878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:33.747892 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:22:33.747977 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:33.747980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:33.748052 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:33.748171 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:22:33.748609 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.749361 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.750680 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:33.750993 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:33.751098 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:33.751559 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:22:33.752165 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:22:33.752310 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:33.752801 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 08:22:33.752874 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:22:33.761608 32316 raft_consensus.cc:1261] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 1 FOLLOWER]: Refusing update from remote peer c6899cd724704b5292364923eaf6af59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:33.762650 32683 consensus_queue.cc:1035] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b3bbc7272b4f6983250e43f4ad38fb" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:33.767710 32539 raft_consensus.cc:1261] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 1 FOLLOWER]: Refusing update from remote peer c6899cd724704b5292364923eaf6af59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:33.768518 32683 consensus_queue.cc:1035] T 9cfe7a490e9b4449953ea8802d3e730d P c6899cd724704b5292364923eaf6af59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab91b9963844d86a17605c145dec508" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:33.776968 32694 mvcc.cc:205] Tried to move back new op lower bound from 7316182440996872192 to 7316182440863604736. Current Snapshot: MvccSnapshot[applied={T|T < 7316182440996872192}]
I20260808 08:22:33.788833 31873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:22:33.789033 31873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:22:33.793040 32697 mvcc.cc:205] Tried to move back new op lower bound from 7316182440996872192 to 7316182440863604736. Current Snapshot: MvccSnapshot[applied={T|T < 7316182440996872192}]
I20260808 08:22:33.795864 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:33 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260808 08:22:33.796128 32482 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:33 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:51808
I20260808 08:22:33.796471 31873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55cbf1d9b94048fd9cc6e4373c879aab: Sending DeleteTablet for 3 replicas of tablet 9cfe7a490e9b4449953ea8802d3e730d
I20260808 08:22:33.797689 32482 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:33 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:51808
I20260808 08:22:33.797973 32289 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfe7a490e9b4449953ea8802d3e730d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:33 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:48598
I20260808 08:22:33.799425 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 31936
I20260808 08:22:33.806360 32695 mvcc.cc:205] Tried to move back new op lower bound from 7316182440996872192 to 7316182440863604736. Current Snapshot: MvccSnapshot[applied={T|T < 7316182440996872192}]
I20260808 08:22:33.809258 32707 tablet_replica.cc:330] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: stopping tablet replica
I20260808 08:22:33.809639 32707 raft_consensus.cc:2229] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:33.810060 32707 raft_consensus.cc:2258] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:33.810530 32710 tablet_replica.cc:330] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: stopping tablet replica
I20260808 08:22:33.810844 32710 raft_consensus.cc:2229] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:33.811147 32710 raft_consensus.cc:2258] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:33.812956 32707 ts_tablet_manager.cc:1905] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:22:33.816831 32707 ts_tablet_manager.cc:1918] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:22:33.817018 32707 log.cc:1196] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/9cfe7a490e9b4449953ea8802d3e730d
I20260808 08:22:33.817986 32707 ts_tablet_manager.cc:1939] T 9cfe7a490e9b4449953ea8802d3e730d P 4ab91b9963844d86a17605c145dec508: Deleting consensus metadata
W20260808 08:22:33.818581 31848 connection.cc:535] client connection to 127.12.13.1:45881 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 08:22:33.818586 32710 ts_tablet_manager.cc:1905] T 9cfe7a490e9b4449953ea8802d3e730d P 37b3bbc7272b4f6983250e43f4ad38fb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 08:22:33.818758 31848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 08:22:33.820421 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.820027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:45881 (local address 127.0.0.1:45956)
0808 08:22:33.820216 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.820313 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:45881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 08:22:33.820479 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 32192
W20260808 08:22:33.831250 31848 catalog_manager.cc:4584] TS c6899cd724704b5292364923eaf6af59 (127.12.13.1:45881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cfe7a490e9b4449953ea8802d3e730d: Network error: Client connection negotiation failed: client connection to 127.12.13.1:45881: connect: Connection refused (error 111)
W20260808 08:22:33.831719 31848 catalog_manager.cc:4808] TS 4ab91b9963844d86a17605c145dec508 (127.12.13.3:35197): delete failed for tablet 9cfe7a490e9b4449953ea8802d3e730d because the tablet was not found. No further retry: Not found: Tablet not found: 9cfe7a490e9b4449953ea8802d3e730d
I20260808 08:22:33.834847 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.834484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:45881 (local address 127.0.0.1:45962)
0808 08:22:33.834683 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.834741 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:45881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 08:22:33.843924 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 32394
I20260808 08:22:33.858719 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:33.844256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:42543 (local address 127.0.0.1:48608)
0808 08:22:33.858477 (+ 14221us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:33.858581 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.2:42543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14058}
I20260808 08:22:33.868628 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 31821
I20260808 08:22:33.885771 12340 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 08:22:34.230926 12340 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 08:22:34.231045 12340 mini_ranger.cc:66] Stopping Ranger...
I20260808 08:22:34.586635 12340 mini_ranger.cc:68] Stopped Ranger
2026-08-08 08:22:34.586 UTC [30308] LOG: received smart shutdown request
2026-08-08 08:22:34.592 UTC [30308] LOG: background worker "logical replication launcher" (PID 30326) exited with exit code 1
2026-08-08 08:22:34.592 UTC [30317] LOG: shutting down
2026-08-08 08:22:34.744 UTC [30308] LOG: database system is shut down
2026-08-08T08:22:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54082 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:22:34 dist-test-slave-6ntk krb5kdc[32728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:22:34 dist-test-slave-6ntk krb5kdc[32728](info): set up 2 sockets
Aug 08 08:22:34 dist-test-slave-6ntk krb5kdc[32728](info): commencing operation
krb5kdc: starting...
W20260808 08:22:36.151033 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:22:36.298007 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:39357
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:39357,127.12.13.61:43693,127.12.13.60:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:36.417037 32744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:36.417400 32744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:36.417497 32744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:36.421600 32744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:36.421705 32744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:36.421744 32744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:36.421783 32744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:36.421818 32744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:36.427232 32744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:39357,127.12.13.61:43693,127.12.13.60:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:36.428437 32744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:36.429574 32744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:22:36.438783 32744 server_base.cc:1034] running on GCE node
W20260808 08:22:36.438712 32751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:36.438747 32753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:36.438736 32756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:36.439803 32744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:36.441097 32744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:36.442277 32744 hybrid_clock.cc:648] HybridClock initialized: now 1786177356442243 us; error 49 us; skew 500 ppm
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:36.449687 32744 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:36.450742 32744 webserver.cc:460] Webserver started at http://127.12.13.62:37963/ using document root <none> and password file <none>
I20260808 08:22:36.451368 32744 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:36.451432 32744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:36.451666 32744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:36.453805 32744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "2d8bfcfeea8b99cb2a3bebd85cc032cd"
server_key_iv: "171592c8a4931793d56f0f209f5b7e9b"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.454463 32744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "2d8bfcfeea8b99cb2a3bebd85cc032cd"
server_key_iv: "171592c8a4931793d56f0f209f5b7e9b"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.458426 32744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:22:36.461025 32767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:36.462432 32744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:22:36.462579 32744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "2d8bfcfeea8b99cb2a3bebd85cc032cd"
server_key_iv: "171592c8a4931793d56f0f209f5b7e9b"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.462697 32744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:36.475652 32744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:36.478837 32744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:36.479060 32744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:36.487457 32744 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:39357
I20260808 08:22:36.487486 392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:39357 every 8 connection(s)
I20260808 08:22:36.488502 32744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:36.491494 393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:36.495626 393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.496112 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 32744
I20260808 08:22:36.496245 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:22:36.496567 12340 external_mini_cluster.cc:1428] Setting key 07a1d6d4c0a1b3e10011c1f276ea18e7
WARNING: no policy specified for kudu/127.12.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.61@KRBTEST.COM" created.
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:36.517119 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.498488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:39357 (local address 127.0.0.1:54794)
0808 08:22:36.499151 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.499217 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.500580 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.501408 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.501417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.502045 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.502490 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.502515 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.503704 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.503710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.505039 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.505052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.505769 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.505777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.505976 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.506660 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.506682 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.511467 (+ 4785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.513723 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.513732 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.513744 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.514075 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.514421 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.514426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.514428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.514551 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.515250 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.515256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.515408 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.515587 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.516007 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":112,"thread_start_us":193,"threads_started":1}
I20260808 08:22:36.517119 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.498488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.12.13.62:39357)
0808 08:22:36.499529 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.499539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.499577 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:22:36.500942 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.500947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.501165 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.501354 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.502669 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.503560 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.505185 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.505583 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.505719 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.511656 (+ 5937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.511669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.511682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.511706 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.513558 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.514198 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.514207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.514214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.514320 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.514681 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.514684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.514686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.515117 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.515269 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.515675 (+ 406us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.515950 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":930,"spinlock_wait_cycles":101120,"thread_start_us":203,"threads_started":1}
I20260808 08:22:36.520061 393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.520849 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.520561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42506)
0808 08:22:36.520690 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.520772 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 08:22:36.521059 301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
I20260808 08:22:36.522122 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.521845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42508)
0808 08:22:36.522008 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.522062 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260808 08:22:36.522330 393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:43693: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab.
Entry for principal kudu/127.12.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab.
I20260808 08:22:36.528960 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:43693
--webserver_interface=127.12.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab
--principal=kudu/127.12.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:39357,127.12.13.61:43693,127.12.13.60:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:36.571578 393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } attempt: 1
I20260808 08:22:36.572537 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.572200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42522)
0808 08:22:36.572369 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.572453 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 08:22:36.573408 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.573171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42526)
0808 08:22:36.573287 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.573338 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:22:36.573597 393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:43693: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
I20260808 08:22:36.641822 393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } attempt: 2
I20260808 08:22:36.642908 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.642531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42528)
0808 08:22:36.642746 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.642822 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 08:22:36.643775 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.643558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42530)
0808 08:22:36.643665 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.643714 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:22:36.643973 393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:43693: Network error: Client connection negotiation failed: client connection to 127.12.13.61:43693: connect: Connection refused (error 111)
W20260808 08:22:36.652339 403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:36.652666 403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:36.652755 403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:36.657044 403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:36.657136 403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:36.657174 403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:36.657230 403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:36.657366 403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:36.662950 403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:39357,127.12.13.61:43693,127.12.13.60:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab
--principal=kudu/127.12.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:36.664083 403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:36.665163 403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:36.674379 409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:36.674379 410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:36.674520 413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:36.674578 403 server_base.cc:1034] running on GCE node
I20260808 08:22:36.675439 403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:36.676635 403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:36.677809 403 hybrid_clock.cc:648] HybridClock initialized: now 1786177356677794 us; error 26 us; skew 500 ppm
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:36.680769 403 init.cc:377] Logged in from keytab as kudu/127.12.13.61@KRBTEST.COM (short username kudu)
I20260808 08:22:36.681716 403 webserver.cc:460] Webserver started at http://127.12.13.61:37099/ using document root <none> and password file <none>
I20260808 08:22:36.682394 403 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:36.682495 403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:36.682746 403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:36.684928 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data/instance:
uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "162e35fd67bbfb45e1bd73983aa68410"
server_key_iv: "7be5ab11a3e7b5d6089cbc830ef340ba"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.685510 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal/instance:
uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "162e35fd67bbfb45e1bd73983aa68410"
server_key_iv: "7be5ab11a3e7b5d6089cbc830ef340ba"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.689402 403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:22:36.692034 422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:36.693282 403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:22:36.693480 403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal
uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "162e35fd67bbfb45e1bd73983aa68410"
server_key_iv: "7be5ab11a3e7b5d6089cbc830ef340ba"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.693604 403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:36.707207 403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:36.710702 403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:36.710960 403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:36.719431 403 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.61:43693
I20260808 08:22:36.719461 500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.61:43693 every 8 connection(s)
I20260808 08:22:36.720484 403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
I20260808 08:22:36.723637 501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:36.727272 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 403
I20260808 08:22:36.727380 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-1/wal/instance
I20260808 08:22:36.727674 12340 external_mini_cluster.cc:1428] Setting key 3c041fd74d91d16fcb9759b2108cae3a
I20260808 08:22:36.728163 501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.735139 393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } attempt: 3
WARNING: no policy specified for kudu/127.12.13.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.60@KRBTEST.COM" created.
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:22:36.748651 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.730448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.12.13.62:39357)
0808 08:22:36.730630 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.730636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.731600 (+ 964us) server_negotiation.cc:407] Connection header received
0808 08:22:36.732750 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.732754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.732826 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.732945 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.734384 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.735307 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.737235 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.737619 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.737745 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.743091 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.743095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.743097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.743146 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.745672 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.746447 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.746451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.746453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.746558 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.747088 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.747093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.747094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.747395 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.747496 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.748206 (+ 710us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.748382 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 08:22:36.748970 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.735693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42546)
0808 08:22:36.735931 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.735949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.736035 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.736857 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.736862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.737074 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.737235 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.737248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.738630 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.738633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.739244 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.739254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.739895 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.739899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.740015 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.740488 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.740501 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.742870 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.747205 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.747212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.747217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.747527 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.747859 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.747865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.747868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.747974 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.748520 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.748523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.748617 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.748697 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.748825 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260808 08:22:36.749643 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.735851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.12.13.61:43693)
0808 08:22:36.736319 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.736333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.736359 (+ 26us) server_negotiation.cc:407] Connection header received
0808 08:22:36.736403 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.736409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.736619 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.736838 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.737381 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.738478 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.739403 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.739760 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.739888 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.743072 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.743085 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.743104 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.743164 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.747088 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.747649 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.747655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.747657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.747747 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.748086 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.748091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.748095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.748388 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.749186 (+ 798us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.749292 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.749449 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":203,"threads_started":1}
I20260808 08:22:36.749643 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.730850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:39357 (local address 127.0.0.1:54804)
0808 08:22:36.731401 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.731461 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.732505 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.733123 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.733130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.733556 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.734197 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.734213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.735504 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.735511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.737074 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.737083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.737841 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.737847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.738135 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.739019 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.739049 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.742705 (+ 3656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.745798 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.745813 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.745830 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.746305 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.746677 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.746683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.746688 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.746946 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.747497 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.747508 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.747931 (+ 423us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.748137 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.748509 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1,"thread_start_us":170,"threads_started":1}
I20260808 08:22:36.751638 501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.751910 393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.752643 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.752337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37516)
0808 08:22:36.752523 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.752574 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 08:22:36.753613 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.753185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37530)
0808 08:22:36.753493 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.753552 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260808 08:22:36.753762 393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:33201: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:22:36.761019 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.752172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42560)
0808 08:22:36.752359 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.752374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.752452 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.752747 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.752750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.752941 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.753109 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.753121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.754392 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.754394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.755018 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.755027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.755544 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.755549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.755720 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.756282 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.756295 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.757785 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.759445 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.759447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.759448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.759724 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.759992 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.759994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.759996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.760072 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.760498 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.760500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.760578 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.760662 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.760857 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1}
I20260808 08:22:36.761027 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.752283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.12.13.61:43693)
0808 08:22:36.752491 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.752495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.752510 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:22:36.752555 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.752560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.752616 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.752732 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.753493 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.754273 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.755136 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.755419 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.755533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.757966 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.757970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.757972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.757995 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.759344 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.759826 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.759829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.759831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.759907 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.760152 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.760154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.760155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.760410 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.760491 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.760722 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.760895 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":129}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab.
Entry for principal kudu/127.12.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab.
I20260808 08:22:36.762574 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:33201
--webserver_interface=127.12.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab
--principal=kudu/127.12.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:39357,127.12.13.61:43693,127.12.13.60:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:36.763195 501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.764084 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.763729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37542)
0808 08:22:36.763877 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.763989 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 08:22:36.764739 426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
I20260808 08:22:36.765796 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.765539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37546)
0808 08:22:36.765671 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.765730 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 08:22:36.766007 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:33201: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
I20260808 08:22:36.784963 393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } attempt: 1
I20260808 08:22:36.786008 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.785586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37556)
0808 08:22:36.785750 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.785821 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 08:22:36.786790 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.786576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37562)
0808 08:22:36.786686 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.786733 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:22:36.787017 393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:33201: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
I20260808 08:22:36.815358 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } attempt: 1
I20260808 08:22:36.816531 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.816147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37566)
0808 08:22:36.816354 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.816430 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 08:22:36.817354 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.817101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37574)
0808 08:22:36.817220 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.817276 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 08:22:36.817535 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:33201: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
I20260808 08:22:36.862205 393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } attempt: 2
I20260808 08:22:36.863166 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.862849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37590)
0808 08:22:36.863019 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.863083 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 08:22:36.863943 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.863720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37592)
0808 08:22:36.863831 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.863890 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 08:22:36.864074 393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:33201: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
W20260808 08:22:36.885463 511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:36.885782 511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260808 08:22:36.885727 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } attempt: 2
W20260808 08:22:36.885980 511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 08:22:36.886680 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.886331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37594)
0808 08:22:36.886512 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.886586 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 08:22:36.887444 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.887220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37608)
0808 08:22:36.887327 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.887379 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 08:22:36.887590 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:33201: Network error: Client connection negotiation failed: client connection to 127.12.13.60:33201: connect: Connection refused (error 111)
W20260808 08:22:36.890743 511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:36.890794 511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:36.890815 511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:36.890870 511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:36.890904 511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:36.896433 511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:39357,127.12.13.61:43693,127.12.13.60:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab
--principal=kudu/127.12.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:36.897655 511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:36.898871 511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:36.908175 521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:36.908324 511 server_base.cc:1034] running on GCE node
W20260808 08:22:36.908192 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:36.908560 529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:36.909245 511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:36.910559 511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:36.911736 511 hybrid_clock.cc:648] HybridClock initialized: now 1786177356911715 us; error 39 us; skew 500 ppm
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:36.914863 511 init.cc:377] Logged in from keytab as kudu/127.12.13.60@KRBTEST.COM (short username kudu)
I20260808 08:22:36.915805 511 webserver.cc:460] Webserver started at http://127.12.13.60:37571/ using document root <none> and password file <none>
I20260808 08:22:36.916435 511 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:36.916534 511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:36.916777 511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:36.919004 511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data/instance:
uuid: "62c7da06741d4830b4de079aed85d348"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "678d64448e11d87456ba68252fa44eaf"
server_key_iv: "9e63ff4054d8a804df7e44662604d1b5"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.919593 511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal/instance:
uuid: "62c7da06741d4830b4de079aed85d348"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "678d64448e11d87456ba68252fa44eaf"
server_key_iv: "9e63ff4054d8a804df7e44662604d1b5"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.923441 511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:22:36.926065 539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:36.927501 511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:22:36.927670 511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal
uuid: "62c7da06741d4830b4de079aed85d348"
format_stamp: "Formatted at 2026-08-08 08:22:36 on dist-test-slave-6ntk"
server_key: "678d64448e11d87456ba68252fa44eaf"
server_key_iv: "9e63ff4054d8a804df7e44662604d1b5"
server_key_version: "encryptionkey@0"
I20260808 08:22:36.927802 511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:36.939781 511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:36.943812 511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:36.944041 511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:36.951764 511 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.60:33201
I20260808 08:22:36.951861 619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.60:33201 every 8 connection(s)
I20260808 08:22:36.952831 511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
I20260808 08:22:36.955960 620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:36.959981 620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.960886 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 511
I20260808 08:22:36.961026 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786177179845590-12340-0/minicluster-data/master-2/wal/instance
I20260808 08:22:36.961277 12340 external_mini_cluster.cc:1428] Setting key 4da74e6ea43bf25e7c90420f058e6485
I20260808 08:22:36.963245 393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } attempt: 3
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
I20260808 08:22:36.975945 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.963682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37620)
0808 08:22:36.963807 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.963824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.963917 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.965624 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.965627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.965830 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.965973 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.965980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.967143 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.967148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.967996 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.968008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.968762 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.968765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.969146 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.969644 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.969656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.971436 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.973750 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.973753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.973756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.974067 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.974459 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.974463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.974466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.974558 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.975339 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.975344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.975457 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.975591 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.975743 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 08:22:36.976835 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.962148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.12.13.62:39357)
0808 08:22:36.962289 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.962294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.963233 (+ 939us) server_negotiation.cc:407] Connection header received
0808 08:22:36.964297 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.964300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.964363 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.964461 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.965611 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.966507 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.967872 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.968131 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.968242 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.971200 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.971203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.971206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.971241 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.972913 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.974038 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.974043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.974046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.974151 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.974928 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.974932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.974933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.975216 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.975326 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.976577 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.976727 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 08:22:36.977519 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.963739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.12.13.60:33201)
0808 08:22:36.964061 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.964067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.964098 (+ 31us) server_negotiation.cc:407] Connection header received
0808 08:22:36.964134 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.964139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.964426 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.964663 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.966103 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.967034 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.968122 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.968611 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.968754 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.971598 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.971604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.971615 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.971663 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.973632 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.974207 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.974213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.974215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.974352 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.974661 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.974664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.974666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.975217 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.975352 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.975812 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.976327 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":31744,"thread_start_us":93,"threads_started":1}
I20260808 08:22:36.977525 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.962558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:39357 (local address 127.0.0.1:54810)
0808 08:22:36.963074 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.963124 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.964092 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.964518 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.964527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.965064 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.965479 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.965503 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.966677 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.966679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.967745 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.967755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.968335 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.968339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.968549 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.969210 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.969231 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.970993 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.973040 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.973053 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.973066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.973839 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.974304 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.974310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.974314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.974793 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.976153 (+ 1360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.976160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.976306 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.977062 (+ 756us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.977281 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"spinlock_wait_cycles":1152,"thread_start_us":186,"threads_started":1}
I20260808 08:22:36.977759 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.962561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.12.13.60:33201)
0808 08:22:36.962981 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.963011 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.963045 (+ 34us) server_negotiation.cc:407] Connection header received
0808 08:22:36.964108 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.964139 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.964426 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.964663 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.965605 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.967035 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.968293 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.968680 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.969210 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.972788 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.972791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.972793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.972815 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.975258 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.975774 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.975778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.975779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.975863 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.976194 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.976197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.976199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.976416 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.977286 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.977367 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.977481 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":274,"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":12544,"thread_start_us":138,"threads_started":1}
I20260808 08:22:36.979019 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } attempt: 3
I20260808 08:22:36.979470 620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.982514 393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: Bootstrap starting.
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:22:36.985668 393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:36.986647 393 log.cc:824] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:36.989509 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.979909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.12.13.60:33201)
0808 08:22:36.980406 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.980410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.980424 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:22:36.980458 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.980460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.980501 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.981183 (+ 682us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.981255 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.981749 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.982560 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.982917 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.983043 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.985762 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.985767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.985769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.985796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.987570 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.988193 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.988203 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.988205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.988289 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.988594 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.988597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.988599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.988789 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.988896 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.989243 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.989360 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":399}
I20260808 08:22:36.989509 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.979498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37630)
0808 08:22:36.979764 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.979782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.979877 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.980790 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.980794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.980990 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.981087 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.981093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.981892 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.981899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.982428 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.982436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.983050 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.983053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.983138 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.983675 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.983692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.985602 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.987713 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.987717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.987721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.988082 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.988391 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.988394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.988396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.988488 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.988902 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.988908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.989010 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.989178 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.989330 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260808 08:22:36.991058 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.980271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.12.13.61:43693)
0808 08:22:36.980392 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.980397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.980410 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:22:36.980449 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.980451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.980494 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.980569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.981161 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.981931 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.983001 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.983346 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.983422 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.986512 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.986517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.986522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.986550 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.988534 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.989243 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.989246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.989248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.989327 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.990105 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.990108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.990110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.990307 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.990422 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.990783 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.990913 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 08:22:36.991533 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.979987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:43693 (local address 127.0.0.1:42570)
0808 08:22:36.980092 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.980105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.980179 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.980691 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.980694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.980877 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.981011 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.981019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.982075 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.982078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.982859 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.982870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.983571 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.983574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:36.983679 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.984124 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:36.984139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:36.986074 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:36.988881 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.988884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.988886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.989125 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.989450 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:36.989452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:36.989454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:36.989979 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:36.990472 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:36.990475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:36.990568 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 08:22:36.991236 (+ 668us) client_negotiation.cc:240] Negotiation successful
0808 08:22:36.991378 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":25}
I20260808 08:22:36.992558 393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: No bootstrap required, opened a new log
I20260808 08:22:36.992604 620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:22:36.993229 501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675: Bootstrap starting.
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:36.996276 501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:36.996994 393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:36.997213 393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:36.997270 393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfbf6eca3ec4ce4ad52ceed89cf91c8, State: Initialized, Role: FOLLOWER
I20260808 08:22:36.997303 501 log.cc:824] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:36.997812 393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:36.999308 393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:36.999644 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.980742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.12.13.62:39357)
0808 08:22:36.980858 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.980862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.987534 (+ 6672us) server_negotiation.cc:407] Connection header received
0808 08:22:36.987583 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.987585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.987637 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.987716 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.988657 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.989439 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.990591 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.990893 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.990991 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:36.994984 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:36.994988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:36.994991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:36.995020 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:36.996708 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.997515 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.997520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.997522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.997618 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:36.998148 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:36.998151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:36.998153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:36.998420 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:36.998599 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:36.999315 (+ 716us) server_negotiation.cc:299] Negotiation successful
0808 08:22:36.999497 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 08:22:37.001080 501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675: No bootstrap required, opened a new log
Aug 08 08:22:36 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
I20260808 08:22:37.006304 501 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.006577 501 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:37.006654 501 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f9f97bcae44272a3ccd2a2e3bdc675, State: Initialized, Role: FOLLOWER
I20260808 08:22:37.007256 501 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.007800 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.993112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:33201 (local address 127.0.0.1:37634)
0808 08:22:36.993275 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:36.993292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:36.993377 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:36.998382 (+ 5005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:36.998385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:36.998575 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:36.998688 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:36.998696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.999664 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:36.999666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:37.000483 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:37.000495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.001205 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.001211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:37.001286 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:37.001823 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:37.001838 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:37.003476 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:37.005537 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:37.005539 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:37.005542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:37.005797 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:37.006124 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:37.006127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:37.006128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:37.006205 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:37.007063 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:37.007066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:37.007170 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 08:22:37.007260 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 08:22:37.007619 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 08:22:37.008355 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:36.993270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.12.13.60:33201)
0808 08:22:36.998059 (+ 4789us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:36.998063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:36.998076 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:22:36.998116 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:36.998121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:36.998175 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:36.998271 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:36.998830 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:36.999536 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.000617 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.000921 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.001028 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:37.003620 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:37.003624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:37.003627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:37.003652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:37.005415 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.005951 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.005954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.005956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.006037 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.006291 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.006294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.006295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.006499 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:37.007532 (+ 1033us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:37.008103 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 08:22:37.008227 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4692}
Aug 08 08:22:37 dist-test-slave-6ntk krb5kdc[32728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:22:37.008867 501 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:37.005751 32744 master_runner.cc:186] Error getting master registration for 127.12.13.61:43693: OK, instance_id { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" instance_seqno: 1786177356717005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 08:22:37.010588 32744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 08:22:37.012729 647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:22:37.012813 647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:22:37.013160 620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: Bootstrap starting.
I20260808 08:22:37.014482 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:37.001628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.12.13.61:43693)
0808 08:22:37.001748 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:37.001754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:37.002092 (+ 338us) server_negotiation.cc:407] Connection header received
0808 08:22:37.002140 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:37.002143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:37.002196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:37.002270 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:37.003136 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.003835 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.005795 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.006177 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.006282 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:37.010045 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:37.010049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:37.010052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:37.010081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:37.011845 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.012461 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.012464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.012466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.012556 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.012887 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.012890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.012892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.013143 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:37.013375 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:37.014177 (+ 802us) server_negotiation.cc:299] Negotiation successful
0808 08:22:37.014308 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 08:22:37.016214 620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:37.017187 620 log.cc:824] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:37.017794 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:22:37.017980 403 master_runner.cc:186] Error getting master registration for 127.12.13.60:33201: OK, instance_id { permanent_uuid: "62c7da06741d4830b4de079aed85d348" instance_seqno: 1786177356949540 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 08:22:37.021687 403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 08:22:37.022042 631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.022195 631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.023283 620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: No bootstrap required, opened a new log
W20260808 08:22:37.025727 666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:22:37.025810 666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:22:37.026060 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.026192 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.027726 620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.028013 620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:37.028064 620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7da06741d4830b4de079aed85d348, State: Initialized, Role: FOLLOWER
I20260808 08:22:37.028642 620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.030148 620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:37.032943 680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.033401 680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.039094 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:37.020912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.12.13.62:39357)
0808 08:22:37.021056 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:37.021061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:37.021966 (+ 905us) server_negotiation.cc:407] Connection header received
0808 08:22:37.022092 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:37.022095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:37.022146 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:37.022254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:37.023057 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.023674 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.024798 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.025125 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.025219 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:37.027339 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:37.027343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:37.027347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:37.027375 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:37.033029 (+ 5654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.033662 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.033666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.033668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.033766 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.037305 (+ 3539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.037309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.037311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.037581 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:37.037724 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:37.038778 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0808 08:22:37.038921 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:22:37.040949 511 master_runner.cc:263] No leader master found from master 62c7da06741d4830b4de079aed85d348
I20260808 08:22:37.041038 511 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 08:22:37.042834 694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 08:22:37.045092 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:37.021239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.12.13.60:33201)
0808 08:22:37.021374 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:37.021378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:37.026816 (+ 5438us) server_negotiation.cc:407] Connection header received
0808 08:22:37.027095 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:37.027098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:37.027149 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:37.027202 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:37.029936 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.030604 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.034587 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.034933 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.035056 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:37.036846 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:37.036850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:37.036853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:37.036886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:37.038765 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.043536 (+ 4771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.043540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.043543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.043648 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.043999 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.044004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.044007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.044287 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:37.044411 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:37.044783 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 08:22:37.044918 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 08:22:37.046178 694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:22:37.046999 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:37.020669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.12.13.61:43693)
0808 08:22:37.020782 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:37.020786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:37.026607 (+ 5821us) server_negotiation.cc:407] Connection header received
0808 08:22:37.027344 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:37.027348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:37.027393 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:37.027442 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:37.029064 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.029752 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.033962 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:37.034318 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:37.034403 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:37.043094 (+ 8691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:37.043098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:37.043101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:37.043130 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:37.044893 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.045421 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.045427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.045430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.045527 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:37.045853 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:37.045856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:37.045858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:37.046142 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:37.046278 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:37.046665 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 08:22:37.046799 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 08:22:37.087172 649 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:37.087426 649 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.088646 649 leader_election.cc:288] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dfbf6eca3ec4ce4ad52ceed89cf91c8 (127.12.13.62:39357), 62c7da06741d4830b4de079aed85d348 (127.12.13.60:33201)
I20260808 08:22:37.089380 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" is_pre_election: true
I20260808 08:22:37.089771 349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2f9f97bcae44272a3ccd2a2e3bdc675 in term 0.
I20260808 08:22:37.090278 424 leader_election.cc:302] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dfbf6eca3ec4ce4ad52ceed89cf91c8, b2f9f97bcae44272a3ccd2a2e3bdc675; no voters:
I20260808 08:22:37.090612 649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:22:37.090716 649 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:22:37.090778 649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:37.092259 649 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.093042 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8"
I20260808 08:22:37.093186 349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:37.094662 349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2f9f97bcae44272a3ccd2a2e3bdc675 in term 1.
I20260808 08:22:37.095039 424 leader_election.cc:302] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dfbf6eca3ec4ce4ad52ceed89cf91c8, b2f9f97bcae44272a3ccd2a2e3bdc675; no voters:
I20260808 08:22:37.095994 580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7da06741d4830b4de079aed85d348" is_pre_election: true
I20260808 08:22:37.096398 580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2f9f97bcae44272a3ccd2a2e3bdc675 in term 0.
I20260808 08:22:37.097463 580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7da06741d4830b4de079aed85d348"
I20260808 08:22:37.097594 580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:37.100405 580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2f9f97bcae44272a3ccd2a2e3bdc675 in term 1.
I20260808 08:22:37.101401 649 leader_election.cc:288] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [CANDIDATE]: Term 1 election: Requested vote from peers 7dfbf6eca3ec4ce4ad52ceed89cf91c8 (127.12.13.62:39357), 62c7da06741d4830b4de079aed85d348 (127.12.13.60:33201)
I20260808 08:22:37.101589 649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:37.101692 649 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [term 1 LEADER]: Becoming Leader. State: Replica: b2f9f97bcae44272a3ccd2a2e3bdc675, State: Running, Role: LEADER
I20260808 08:22:37.102125 649 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } }
I20260808 08:22:37.104179 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f9f97bcae44272a3ccd2a2e3bdc675. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.104290 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:37.104771 700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:37.105834 700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:37.117074 580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [term 1 FOLLOWER]: Refusing update from remote peer b2f9f97bcae44272a3ccd2a2e3bdc675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:37.117813 697 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:37.122398 680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f9f97bcae44272a3ccd2a2e3bdc675. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.122539 680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.123720 349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [term 1 FOLLOWER]: Refusing update from remote peer b2f9f97bcae44272a3ccd2a2e3bdc675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:37.124423 697 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:37.128998 631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f9f97bcae44272a3ccd2a2e3bdc675. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.129154 631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.139532 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.139736 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:37.141183 631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.141330 631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.141760 701 mvcc.cc:205] Tried to move back new op lower bound from 7316182454743064576 to 7316182454693867520. Current Snapshot: MvccSnapshot[applied={T|T < 7316182454743064576}]
I20260808 08:22:37.143280 706 mvcc.cc:205] Tried to move back new op lower bound from 7316182454743064576 to 7316182454693867520. Current Snapshot: MvccSnapshot[applied={T|T < 7316182454743064576}]
I20260808 08:22:37.144560 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.144707 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:37.148362 680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfbf6eca3ec4ce4ad52ceed89cf91c8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 39357 } } peers { permanent_uuid: "b2f9f97bcae44272a3ccd2a2e3bdc675" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 43693 } } peers { permanent_uuid: "62c7da06741d4830b4de079aed85d348" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 33201 } } }
I20260808 08:22:37.148497 680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:22:37.149806 704 mvcc.cc:205] Tried to move back new op lower bound from 7316182454743064576 to 7316182454693867520. Current Snapshot: MvccSnapshot[applied={T|T < 7316182454743064576}]
I20260808 08:22:37.154104 700 catalog_manager.cc:1347] Generated new cluster ID: 7f1bbcc56b4e41f39463a8c9b87d3a8c
I20260808 08:22:37.154189 700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:22:37.189237 700 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:22:37.190645 700 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:22:37.211865 700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f9f97bcae44272a3ccd2a2e3bdc675: Generated new TSK 0
I20260808 08:22:37.216153 700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:22:38.014429 647 catalog_manager.cc:1259] Loaded cluster ID: 7f1bbcc56b4e41f39463a8c9b87d3a8c
I20260808 08:22:38.014523 647 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: loading cluster ID for follower catalog manager: success
I20260808 08:22:38.016341 647 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: acquiring CA information for follower catalog manager: success
I20260808 08:22:38.017637 647 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7dfbf6eca3ec4ce4ad52ceed89cf91c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:22:38.047735 694 catalog_manager.cc:1259] Loaded cluster ID: 7f1bbcc56b4e41f39463a8c9b87d3a8c
I20260808 08:22:38.047832 694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: loading cluster ID for follower catalog manager: success
I20260808 08:22:38.049669 694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: acquiring CA information for follower catalog manager: success
I20260808 08:22:38.050998 694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62c7da06741d4830b4de079aed85d348: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:22:38.294689 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 32744
I20260808 08:22:38.309377 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 403
I20260808 08:22:38.324450 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 511
2026-08-08T08:22:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3465 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:22:38 dist-test-slave-6ntk krb5kdc[719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:22:38 dist-test-slave-6ntk krb5kdc[719](info): set up 2 sockets
Aug 08 08:22:38 dist-test-slave-6ntk krb5kdc[719](info): commencing operation
krb5kdc: starting...
W20260808 08:22:39.578311 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:22:39 dist-test-slave-6ntk krb5kdc[719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:22:39.715289 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46497
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:39.831794 742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:39.832134 742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:39.832229 742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:39.836293 742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:39.836407 742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:39.836443 742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:39.836496 742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:39.836529 742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:39.842082 742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:46497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:39.843232 742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:39.844311 742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:39.853461 753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:39.853523 755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:39.853540 742 server_base.cc:1034] running on GCE node
W20260808 08:22:39.853469 752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:39.854583 742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:39.855737 742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:39.856938 742 hybrid_clock.cc:648] HybridClock initialized: now 1786177359856908 us; error 42 us; skew 500 ppm
Aug 08 08:22:39 dist-test-slave-6ntk krb5kdc[719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:39.859769 742 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:39.860693 742 webserver.cc:460] Webserver started at http://127.12.13.62:40093/ using document root <none> and password file <none>
I20260808 08:22:39.861285 742 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:39.861382 742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:39.861632 742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:39.863849 742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "02e277c8c72b41379edda3a4b3cd57ed"
format_stamp: "Formatted at 2026-08-08 08:22:39 on dist-test-slave-6ntk"
server_key: "397169cfa0f548a7003e54535b601aac"
server_key_iv: "dca17ae353a7018b34bc708579ef73b6"
server_key_version: "encryptionkey@0"
I20260808 08:22:39.864444 742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "02e277c8c72b41379edda3a4b3cd57ed"
format_stamp: "Formatted at 2026-08-08 08:22:39 on dist-test-slave-6ntk"
server_key: "397169cfa0f548a7003e54535b601aac"
server_key_iv: "dca17ae353a7018b34bc708579ef73b6"
server_key_version: "encryptionkey@0"
I20260808 08:22:39.868312 742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:22:39.870957 765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:39.872195 742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:22:39.872388 742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "02e277c8c72b41379edda3a4b3cd57ed"
format_stamp: "Formatted at 2026-08-08 08:22:39 on dist-test-slave-6ntk"
server_key: "397169cfa0f548a7003e54535b601aac"
server_key_iv: "dca17ae353a7018b34bc708579ef73b6"
server_key_version: "encryptionkey@0"
I20260808 08:22:39.872546 742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:39.887712 742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:39.891304 742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:39.891538 742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:39.899662 742 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:46497
I20260808 08:22:39.899670 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:46497 every 8 connection(s)
I20260808 08:22:39.900766 742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:39.903102 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 742
I20260808 08:22:39.903231 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:22:39.903599 12340 external_mini_cluster.cc:1428] Setting key 135b43e58adf628d2a147e79714a3086
I20260808 08:22:39.904035 849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:39.911319 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed: Bootstrap starting.
Aug 08 08:22:39 dist-test-slave-6ntk krb5kdc[719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:39.914238 849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:39.915241 849 log.cc:824] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:39.917724 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed: No bootstrap required, opened a new log
I20260808 08:22:39.919790 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:39.905040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.12.13.62:46497)
0808 08:22:39.905519 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:39.905537 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:39.905576 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:22:39.906382 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:39.906401 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:39.906772 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:39.907071 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:39.908042 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:39.909171 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:39.910426 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:39.910931 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:39.911064 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:39.914230 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:39.914237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:39.914257 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:39.914295 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:39.916439 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:39.916902 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:39.916908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:39.916914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:39.917053 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:39.917404 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:39.917408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:39.917409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:39.917978 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:39.918188 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:39.918459 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 08:22:39.918771 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":176,"threads_started":1}
I20260808 08:22:39.922260 849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e277c8c72b41379edda3a4b3cd57ed" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46497 } }
I20260808 08:22:39.922565 849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:39.922657 849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e277c8c72b41379edda3a4b3cd57ed, State: Initialized, Role: FOLLOWER
I20260808 08:22:39.923065 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:22:39.923179 849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e277c8c72b41379edda3a4b3cd57ed" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46497 } }
I20260808 08:22:39.923359 849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:22:39.923449 849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:22:39.923552 849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:39.924643 849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e277c8c72b41379edda3a4b3cd57ed" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46497 } }
I20260808 08:22:39.925016 849 leader_election.cc:302] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02e277c8c72b41379edda3a4b3cd57ed; no voters:
I20260808 08:22:39.925370 849 leader_election.cc:288] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:22:39.925498 858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:39.925771 858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [term 1 LEADER]: Becoming Leader. State: Replica: 02e277c8c72b41379edda3a4b3cd57ed, State: Running, Role: LEADER
I20260808 08:22:39.926177 858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e277c8c72b41379edda3a4b3cd57ed" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46497 } }
I20260808 08:22:39.926699 849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:39.927855 860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02e277c8c72b41379edda3a4b3cd57ed. Latest consensus state: current_term: 1 leader_uuid: "02e277c8c72b41379edda3a4b3cd57ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e277c8c72b41379edda3a4b3cd57ed" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46497 } } }
I20260808 08:22:39.928001 860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:39.928161 859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02e277c8c72b41379edda3a4b3cd57ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e277c8c72b41379edda3a4b3cd57ed" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46497 } } }
I20260808 08:22:39.928344 859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:39.928699 871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:39.931712 871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:39.937493 871 catalog_manager.cc:1347] Generated new cluster ID: 8bf0bbe551ae41658716e052f34dc8f9
I20260808 08:22:39.937572 871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:22:39.950486 871 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:22:39.951467 871 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:22:39.957805 871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02e277c8c72b41379edda3a4b3cd57ed: Generated new TSK 0
I20260808 08:22:39.958726 871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:22:39.979418 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 742
2026-08-08T08:22:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1652 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:22:40 dist-test-slave-6ntk krb5kdc[889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:22:40 dist-test-slave-6ntk krb5kdc[889](info): set up 2 sockets
Aug 08 08:22:40 dist-test-slave-6ntk krb5kdc[889](info): commencing operation
krb5kdc: starting...
W20260808 08:22:41.236109 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:22:41 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:22:41.375204 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43219
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:41.493657 904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:41.494065 904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:41.494155 904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:41.498322 904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:41.498430 904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:41.498466 904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:41.498507 904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:41.498540 904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:41.504030 904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:41.505306 904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:41.506505 904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:41.515547 914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:41.515638 904 server_base.cc:1034] running on GCE node
W20260808 08:22:41.515547 911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:41.515822 916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:41.516418 904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:41.517650 904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:41.518909 904 hybrid_clock.cc:648] HybridClock initialized: now 1786177361518912 us; error 67 us; skew 500 ppm
Aug 08 08:22:41 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:41.527369 904 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:41.528514 904 webserver.cc:460] Webserver started at http://127.12.13.62:32869/ using document root <none> and password file <none>
I20260808 08:22:41.529112 904 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:41.529178 904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:41.529374 904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:41.531718 904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "84807906cc0a49eb97e47df9fed7b8e8"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "39eb98a459f972c185320a762cb9e6b9"
server_key_iv: "95f46c134344e59a65ff927dbf11decf"
server_key_version: "encryptionkey@0"
I20260808 08:22:41.532259 904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "84807906cc0a49eb97e47df9fed7b8e8"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "39eb98a459f972c185320a762cb9e6b9"
server_key_iv: "95f46c134344e59a65ff927dbf11decf"
server_key_version: "encryptionkey@0"
I20260808 08:22:41.536427 904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:22:41.539067 928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:41.540429 904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:22:41.540565 904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "84807906cc0a49eb97e47df9fed7b8e8"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "39eb98a459f972c185320a762cb9e6b9"
server_key_iv: "95f46c134344e59a65ff927dbf11decf"
server_key_version: "encryptionkey@0"
I20260808 08:22:41.540679 904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:41.562943 904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:41.566318 904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:41.566502 904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:41.573999 904 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:43219
I20260808 08:22:41.574321 1023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:43219 every 8 connection(s)
I20260808 08:22:41.575101 904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:41.578151 1024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:41.583760 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 904
I20260808 08:22:41.583935 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:22:41.584235 12340 external_mini_cluster.cc:1428] Setting key 13c1b28e73d358ebaf18205c0693cc93
I20260808 08:22:41.584383 1024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap starting.
I20260808 08:22:41.587270 1024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:41.588310 1024 log.cc:824] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:41.590633 1024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: No bootstrap required, opened a new log
Aug 08 08:22:41 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:41.595171 1024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:41.595451 1024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:41.595536 1024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84807906cc0a49eb97e47df9fed7b8e8, State: Initialized, Role: FOLLOWER
I20260808 08:22:41.595991 1024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:41.596158 1024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:22:41.596251 1024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:22:41.596357 1024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:41.597311 1024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:41.597693 1024 leader_election.cc:302] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84807906cc0a49eb97e47df9fed7b8e8; no voters:
I20260808 08:22:41.598104 1024 leader_election.cc:288] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:22:41.598184 1032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:41.598435 1032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 LEADER]: Becoming Leader. State: Replica: 84807906cc0a49eb97e47df9fed7b8e8, State: Running, Role: LEADER
I20260808 08:22:41.598805 1032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:41.599521 1024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:41.599887 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:41.585640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.12.13.62:43219)
0808 08:22:41.586134 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:41.586151 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:41.586190 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:22:41.586834 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:41.586852 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:41.587287 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:41.587638 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:41.588610 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:41.589624 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:41.590818 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:41.591279 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:41.591405 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:41.594275 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:41.594280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:41.594294 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:41.594321 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:41.596394 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:41.596855 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:41.596862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:41.596868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:41.596996 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:41.597293 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:41.597297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:41.597299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:41.597805 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:41.598073 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:41.598751 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 08:22:41.599069 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":203,"threads_started":1}
I20260808 08:22:41.600620 1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84807906cc0a49eb97e47df9fed7b8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } } }
I20260808 08:22:41.600767 1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:41.601089 1034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84807906cc0a49eb97e47df9fed7b8e8. Latest consensus state: current_term: 1 leader_uuid: "84807906cc0a49eb97e47df9fed7b8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } } }
I20260808 08:22:41.601186 1034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:41.601257 1049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:41.604483 1049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:41.610885 1049 catalog_manager.cc:1347] Generated new cluster ID: 8d3c6fadf65a40878fc39ba121af03ca
I20260808 08:22:41.610963 1049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:22:41.631338 1049 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:22:41.632548 1049 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:22:41.642247 1049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Generated new TSK 0
I20260808 08:22:41.643079 1049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:22:41.681094 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43219
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:41.802996 1069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:41.803279 1069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:41.803324 1069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:41.807417 1069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:41.807485 1069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:41.807545 1069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:22:41.813102 1069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:41.814317 1069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:41.815440 1069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:41.826195 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:41.826195 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:41.826351 1069 server_base.cc:1034] running on GCE node
W20260808 08:22:41.826208 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:41.826941 1069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:41.827715 1069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:41.828928 1069 hybrid_clock.cc:648] HybridClock initialized: now 1786177361828911 us; error 26 us; skew 500 ppm
Aug 08 08:22:41 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:41.831910 1069 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:22:41.832854 1069 webserver.cc:460] Webserver started at http://127.12.13.1:41833/ using document root <none> and password file <none>
I20260808 08:22:41.833513 1069 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:41.833617 1069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:41.833909 1069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:41.836100 1069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "45d98465adfd4ba8a4449c6bf01f226e"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "3ea9704129ee6326f6889620eb3c3882"
server_key_iv: "b677ab56d2bed6772f1dc07ef6ba3c10"
server_key_version: "encryptionkey@0"
I20260808 08:22:41.836715 1069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "45d98465adfd4ba8a4449c6bf01f226e"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "3ea9704129ee6326f6889620eb3c3882"
server_key_iv: "b677ab56d2bed6772f1dc07ef6ba3c10"
server_key_version: "encryptionkey@0"
I20260808 08:22:41.840929 1069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:22:41.843770 1089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:41.845299 1069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:22:41.845444 1069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "45d98465adfd4ba8a4449c6bf01f226e"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "3ea9704129ee6326f6889620eb3c3882"
server_key_iv: "b677ab56d2bed6772f1dc07ef6ba3c10"
server_key_version: "encryptionkey@0"
I20260808 08:22:41.845569 1069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:41.864238 1069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:41.867511 1069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:41.867738 1069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:41.868422 1069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:41.869328 1069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:41.869387 1069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:41.869462 1069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:41.869494 1069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:41.879240 1069 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:44997
I20260808 08:22:41.879310 1267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:44997 every 8 connection(s)
I20260808 08:22:41.880277 1069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:22:41.889350 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 1069
I20260808 08:22:41.889506 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:22:41.889830 12340 external_mini_cluster.cc:1428] Setting key 14835a6b03c4490cdca2bc0ac11612a8
Aug 08 08:22:41 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:41.895728 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:41.882342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:57991 (local address 127.12.13.62:43219)
0808 08:22:41.882505 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:41.882508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:41.883401 (+ 893us) server_negotiation.cc:407] Connection header received
0808 08:22:41.884846 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:41.884850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:41.884900 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:41.884977 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:41.886369 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:41.886905 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:41.888163 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:41.888395 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:41.888477 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:41.891668 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:41.891671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:41.891674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:41.891701 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:41.893407 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:41.894017 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:41.894020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:41.894022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:41.894116 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:41.894536 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:41.894540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:41.894542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:41.894739 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:41.894852 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:41.895422 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 08:22:41.895560 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 08:22:41.896625 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:41.882738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:57991)
0808 08:22:41.883212 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:41.883268 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:41.884625 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:41.885166 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:41.885173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:41.885557 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:41.886216 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:41.886233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:41.887027 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:41.887030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:41.888047 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:41.888057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:41.888546 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:41.888549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:41.888741 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:41.889607 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:41.889634 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:41.891497 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:41.893536 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:41.893540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:41.893554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:41.893823 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:41.894226 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:41.894229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:41.894231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:41.894419 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:41.894852 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:41.894860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:41.895163 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 08:22:41.895361 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 08:22:41.895727 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20260808 08:22:41.898028 1268 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43219
I20260808 08:22:41.898327 1268 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:41.899014 1268 heartbeater.cc:511] Master 127.12.13.62:43219 requested a full tablet report, sending...
I20260808 08:22:41.901029 957 ts_manager.cc:194] Registered new tserver with Master: 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997)
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:22:41.902657 957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:57991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:22:41.925064 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43219
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:42.048246 1281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:42.048650 1281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:42.048740 1281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:42.053814 1281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:42.053939 1281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:42.054018 1281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:22:42.059562 1281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:42.060698 1281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:42.061745 1281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:42.072232 1289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:42.072216 1288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:42.072216 1291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:42.072826 1281 server_base.cc:1034] running on GCE node
I20260808 08:22:42.073297 1281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:42.073997 1281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:42.075219 1281 hybrid_clock.cc:648] HybridClock initialized: now 1786177362075147 us; error 28 us; skew 500 ppm
Aug 08 08:22:42 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177362, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:42.078285 1281 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:22:42.079418 1281 webserver.cc:460] Webserver started at http://127.12.13.2:44099/ using document root <none> and password file <none>
I20260808 08:22:42.080116 1281 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:42.080197 1281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:42.080478 1281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:42.083310 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "7abf678e7bdb4c14a8e86982cf95a238"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "637584bcd970d14266cb6b5f3fe51df9"
server_key_iv: "da05daad97a0dfafc5c0a7b6bcba715d"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.084321 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "7abf678e7bdb4c14a8e86982cf95a238"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "637584bcd970d14266cb6b5f3fe51df9"
server_key_iv: "da05daad97a0dfafc5c0a7b6bcba715d"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.089068 1281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:22:42.092597 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.094105 1281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:22:42.094249 1281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "7abf678e7bdb4c14a8e86982cf95a238"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "637584bcd970d14266cb6b5f3fe51df9"
server_key_iv: "da05daad97a0dfafc5c0a7b6bcba715d"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.094369 1281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:42.109447 1281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:42.112964 1281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:42.113178 1281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:42.113844 1281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:42.114840 1281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:42.114892 1281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.114965 1281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:42.115007 1281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.125025 1281 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:39597
I20260808 08:22:42.125043 1472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:39597 every 8 connection(s)
I20260808 08:22:42.126168 1281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:22:42.133219 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 1281
I20260808 08:22:42.133343 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:22:42.133634 12340 external_mini_cluster.cc:1428] Setting key 495fae96f35afb684ce1417515cf37d3
Aug 08 08:22:42 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177362, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:42.142891 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.128209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:35643 (local address 127.12.13.62:43219)
0808 08:22:42.128438 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.128444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.129516 (+ 1072us) server_negotiation.cc:407] Connection header received
0808 08:22:42.130979 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.130982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.131035 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.131119 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:42.132336 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.132915 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.134602 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.134899 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.135018 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.138380 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:42.138384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:42.138387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:42.138413 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:42.140349 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:42.140926 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:42.140933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:42.140938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:42.141050 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:42.141430 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:42.141440 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:42.141460 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:42.141742 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:42.141868 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.142558 (+ 690us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.142712 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260808 08:22:42.143761 1476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.128769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:35643)
0808 08:22:42.129333 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.129385 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.130745 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.131289 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.131296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.131660 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:42.132175 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.132188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.133063 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.133066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.134460 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.134473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.135089 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.135096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.135320 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.136054 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:42.136081 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:42.138115 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:42.140483 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:42.140488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:42.140507 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:42.140780 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:42.141152 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:42.141155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:42.141156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:42.141306 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:42.141897 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:42.141907 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:42.142277 (+ 370us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.142486 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.142834 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":149,"threads_started":1}
I20260808 08:22:42.145212 1473 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43219
I20260808 08:22:42.145498 1473 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:22:42.146142 1473 heartbeater.cc:511] Master 127.12.13.62:43219 requested a full tablet report, sending...
I20260808 08:22:42.147295 957 ts_manager.cc:194] Registered new tserver with Master: 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
I20260808 08:22:42.147998 957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:35643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:22:42.165474 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43219
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:42.287878 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:42.288192 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:42.288239 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:42.292601 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:42.292665 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:42.292725 1479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:22:42.298324 1479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:42.299502 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:42.300616 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:42.310822 1494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:42.310774 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:42.310753 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:42.311131 1479 server_base.cc:1034] running on GCE node
I20260808 08:22:42.311854 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:42.312567 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:42.313822 1479 hybrid_clock.cc:648] HybridClock initialized: now 1786177362313800 us; error 36 us; skew 500 ppm
Aug 08 08:22:42 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177362, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:42.316814 1479 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:22:42.317662 1479 webserver.cc:460] Webserver started at http://127.12.13.3:41661/ using document root <none> and password file <none>
I20260808 08:22:42.318288 1479 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:42.318356 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:42.318528 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:22:42.320626 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "c3db1b5b143b4d5f831a65c805f446ee"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "2a62672e15e73e6c6d72b40d0eec0ea8"
server_key_iv: "fd30229ef665ed11b641890e8ceea788"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.321163 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "c3db1b5b143b4d5f831a65c805f446ee"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "2a62672e15e73e6c6d72b40d0eec0ea8"
server_key_iv: "fd30229ef665ed11b641890e8ceea788"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.325116 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:22:42.327786 1505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.329293 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:22:42.329463 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "c3db1b5b143b4d5f831a65c805f446ee"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "2a62672e15e73e6c6d72b40d0eec0ea8"
server_key_iv: "fd30229ef665ed11b641890e8ceea788"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.329600 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:42.340701 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:42.343842 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:42.344043 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:42.344698 1479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:42.345554 1479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:42.345602 1479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.345706 1479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:42.345743 1479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.355392 1479 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:36769
I20260808 08:22:42.355432 1670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:36769 every 8 connection(s)
I20260808 08:22:42.356391 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:22:42.363055 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 1479
I20260808 08:22:42.363196 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:22:42.363497 12340 external_mini_cluster.cc:1428] Setting key 00484d043fcd144647589e2724c62482
Aug 08 08:22:42 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177362, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:42.371481 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.358375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:54985 (local address 127.12.13.62:43219)
0808 08:22:42.358575 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.358580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.359496 (+ 916us) server_negotiation.cc:407] Connection header received
0808 08:22:42.360862 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.360865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.360917 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.360995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:42.362318 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.362859 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.364464 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.364793 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.364881 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.368042 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:42.368045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:42.368047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:42.368073 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:42.369381 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:42.370026 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:42.370029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:42.370030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:42.370111 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:42.370470 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:42.370472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:42.370474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:42.370656 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:42.370733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.371252 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.371358 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 08:22:42.372377 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.358828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:54985)
0808 08:22:42.359317 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.359370 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.360644 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.361164 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.361171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.361573 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:42.362159 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.362174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.363004 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.363008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.364309 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.364318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.364962 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.364965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.365156 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.365931 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:42.365957 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:42.367872 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:42.369498 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:42.369503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:42.369517 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:42.369863 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:42.370223 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:42.370226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:42.370227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:42.370376 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:42.370772 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:42.370777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:42.371027 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.371201 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.371496 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260808 08:22:42.373641 1671 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43219
I20260808 08:22:42.373951 1671 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:42.374485 1671 heartbeater.cc:511] Master 127.12.13.62:43219 requested a full tablet report, sending...
I20260808 08:22:42.375582 957 ts_manager.cc:194] Registered new tserver with Master: c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769)
I20260808 08:22:42.376163 957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:54985
I20260808 08:22:42.378343 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:22:42.388226 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.380423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.12.13.62:43219)
0808 08:22:42.380547 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.380554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.380777 (+ 223us) server_negotiation.cc:407] Connection header received
0808 08:22:42.380889 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.380891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.380930 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.380998 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:42.382019 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.382526 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.383427 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.383626 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.383701 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.385120 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:42.385123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:42.385126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:42.385147 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:42.386656 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:42.387085 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:42.387087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:42.387089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:42.387161 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:42.387426 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:42.387429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:42.387430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:42.387636 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:42.387741 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.387999 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.388094 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:22:42.391585 957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:22:42.394666 957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:22:42.409823 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.404352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:39597 (local address 127.0.0.1:48610)
0808 08:22:42.405623 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.405638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.405721 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.406429 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.406432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.406452 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:42.406547 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.406552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.407644 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.407650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.408429 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.408438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.409247 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.409250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.409376 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.409417 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.409551 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.409672 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260808 08:22:42.409832 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.404505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.12.13.2:39597)
0808 08:22:42.406014 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.406027 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.406050 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:22:42.406119 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.406123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.406284 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.406428 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:42.406669 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.407513 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.408554 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.409120 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.409247 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.409405 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.409581 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.409672 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":110,"threads_started":1}
I20260808 08:22:42.410296 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.403697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36769 (local address 127.0.0.1:50798)
0808 08:22:42.404491 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.404541 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.404690 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.405331 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.405336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.405369 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:42.405520 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.405528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.407020 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.407026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.408484 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.408495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.409828 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.409832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.409945 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.409969 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.410056 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.410127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":105,"spinlock_wait_cycles":6656,"thread_start_us":85,"threads_started":1}
I20260808 08:22:42.410797 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.403858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.12.13.3:36769)
0808 08:22:42.404218 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.404235 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.404829 (+ 594us) server_negotiation.cc:407] Connection header received
0808 08:22:42.404936 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.404940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.405175 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.405339 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:42.405663 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.406878 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.408624 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.409536 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.409638 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.409764 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.410518 (+ 754us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.410612 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":132,"threads_started":1}
I20260808 08:22:42.411011 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.404664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.12.13.1:44997)
0808 08:22:42.405139 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.405147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.405170 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:22:42.405246 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.405251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.405438 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.405585 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:42.406065 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.406936 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.409356 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.410198 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.410296 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.410448 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.410761 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.410873 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":65,"threads_started":1}
I20260808 08:22:42.411186 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.403907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:44997 (local address 127.0.0.1:38566)
0808 08:22:42.404552 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.404567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.404690 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.405731 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.405734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.405750 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:42.405824 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.405829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.407082 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.407085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.409204 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.409215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.410511 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.410514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.410601 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.410622 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.410713 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.410776 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":779,"spinlock_wait_cycles":4224,"thread_start_us":93,"threads_started":1}
I20260808 08:22:42.413625 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a (DEFAULT_TABLE table=test-table [id=25433e3a41944862ba437addbd4f12e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:42.413606 1167 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a (DEFAULT_TABLE table=test-table [id=25433e3a41944862ba437addbd4f12e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:42.413628 1369 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a (DEFAULT_TABLE table=test-table [id=25433e3a41944862ba437addbd4f12e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:42.414788 1167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4453a9cb04491fa92cee8ba2d35c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:42.414822 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4453a9cb04491fa92cee8ba2d35c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:42.414822 1369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4453a9cb04491fa92cee8ba2d35c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:42.422724 1698 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap starting.
I20260808 08:22:42.422842 1700 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Bootstrap starting.
I20260808 08:22:42.424813 1698 tablet_bootstrap.cc:654] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:42.425293 1700 tablet_bootstrap.cc:654] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:42.425428 1701 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Bootstrap starting.
I20260808 08:22:42.425726 1698 log.cc:824] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:42.426182 1700 log.cc:824] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:42.427698 1701 tablet_bootstrap.cc:654] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:42.428093 1698 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: No bootstrap required, opened a new log
I20260808 08:22:42.428257 1700 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: No bootstrap required, opened a new log
I20260808 08:22:42.428329 1698 ts_tablet_manager.cc:1397] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:22:42.428472 1700 ts_tablet_manager.cc:1397] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:22:42.428996 1701 log.cc:824] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:42.431077 1701 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: No bootstrap required, opened a new log
I20260808 08:22:42.431265 1701 ts_tablet_manager.cc:1397] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:22:42.432334 1700 raft_consensus.cc:348] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.432336 1698 raft_consensus.cc:348] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.432621 1698 raft_consensus.cc:374] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:42.432741 1700 raft_consensus.cc:374] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:42.432770 1698 raft_consensus.cc:729] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d98465adfd4ba8a4449c6bf01f226e, State: Initialized, Role: FOLLOWER
I20260808 08:22:42.432854 1700 raft_consensus.cc:729] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abf678e7bdb4c14a8e86982cf95a238, State: Initialized, Role: FOLLOWER
I20260808 08:22:42.433385 1698 consensus_queue.cc:260] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.433458 1700 consensus_queue.cc:260] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.434209 1268 heartbeater.cc:503] Master 127.12.13.62:43219 was elected leader, sending a full tablet report...
I20260808 08:22:42.434530 1698 ts_tablet_manager.cc:1428] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260808 08:22:42.435259 1700 ts_tablet_manager.cc:1428] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 08:22:42.435324 1701 raft_consensus.cc:348] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.435657 1473 heartbeater.cc:503] Master 127.12.13.62:43219 was elected leader, sending a full tablet report...
I20260808 08:22:42.435684 1701 raft_consensus.cc:374] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:42.435738 1701 raft_consensus.cc:729] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3db1b5b143b4d5f831a65c805f446ee, State: Initialized, Role: FOLLOWER
I20260808 08:22:42.436188 1701 consensus_queue.cc:260] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.437105 1701 ts_tablet_manager.cc:1428] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260808 08:22:42.437326 1671 heartbeater.cc:503] Master 127.12.13.62:43219 was elected leader, sending a full tablet report...
W20260808 08:22:42.607564 1672 tablet.cc:2369] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:22:42.607893 1708 raft_consensus.cc:482] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:42.608135 1708 raft_consensus.cc:504] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.609330 1708 leader_election.cc:288] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
I20260808 08:22:42.613796 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.609736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:51583 (local address 127.12.13.2:39597)
0808 08:22:42.609930 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.609935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.610352 (+ 417us) server_negotiation.cc:407] Connection header received
0808 08:22:42.610525 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.610528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.610579 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.610640 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:42.610998 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.611540 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.612795 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.613189 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.613271 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.613377 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.613577 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.613658 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:22:42.613799 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.609686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:50379 (local address 127.12.13.3:36769)
0808 08:22:42.609897 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.609905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.610308 (+ 403us) server_negotiation.cc:407] Connection header received
0808 08:22:42.610524 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.610527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.610585 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.610716 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:42.611069 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.611571 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.612791 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.613205 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.613332 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.613395 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.613609 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.613687 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:22:42.613822 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.609636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36769 (local address 127.12.13.1:50379)
0808 08:22:42.610199 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.610212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.610405 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.610736 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.610739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.610760 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:42.610915 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.610920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.611719 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.611722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.612664 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.612673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.613357 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.613360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.613464 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.613482 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.613575 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.613670 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":142,"spinlock_wait_cycles":52608,"thread_start_us":177,"threads_started":1}
I20260808 08:22:42.613858 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.609632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:39597 (local address 127.12.13.1:51583)
0808 08:22:42.610108 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.610129 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:42.610401 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:42.610698 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:42.610704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:42.610724 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:42.610877 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.610884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.611665 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.611670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.612660 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:42.612671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.613327 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.613331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:42.613435 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.613466 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 08:22:42.613529 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 08:22:42.613646 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":50,"spinlock_wait_cycles":3840,"thread_start_us":153,"threads_started":1}
I20260808 08:22:42.614668 1606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4453a9cb04491fa92cee8ba2d35c6a" candidate_uuid: "45d98465adfd4ba8a4449c6bf01f226e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3db1b5b143b4d5f831a65c805f446ee" is_pre_election: true
I20260808 08:22:42.614655 1394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4453a9cb04491fa92cee8ba2d35c6a" candidate_uuid: "45d98465adfd4ba8a4449c6bf01f226e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf678e7bdb4c14a8e86982cf95a238" is_pre_election: true
I20260808 08:22:42.615047 1394 raft_consensus.cc:2454] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d98465adfd4ba8a4449c6bf01f226e in term 0.
I20260808 08:22:42.615047 1606 raft_consensus.cc:2454] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d98465adfd4ba8a4449c6bf01f226e in term 0.
I20260808 08:22:42.615595 1090 leader_election.cc:302] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e, 7abf678e7bdb4c14a8e86982cf95a238; no voters:
I20260808 08:22:42.615914 1708 raft_consensus.cc:2781] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:22:42.616021 1708 raft_consensus.cc:482] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:22:42.616087 1708 raft_consensus.cc:3037] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:42.617214 1708 raft_consensus.cc:504] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.617614 1708 leader_election.cc:288] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 election: Requested vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
I20260808 08:22:42.618208 1606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4453a9cb04491fa92cee8ba2d35c6a" candidate_uuid: "45d98465adfd4ba8a4449c6bf01f226e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3db1b5b143b4d5f831a65c805f446ee"
I20260808 08:22:42.618268 1394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4453a9cb04491fa92cee8ba2d35c6a" candidate_uuid: "45d98465adfd4ba8a4449c6bf01f226e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf678e7bdb4c14a8e86982cf95a238"
I20260808 08:22:42.618369 1606 raft_consensus.cc:3037] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:42.618464 1394 raft_consensus.cc:3037] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:42.619599 1606 raft_consensus.cc:2454] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d98465adfd4ba8a4449c6bf01f226e in term 1.
I20260808 08:22:42.620003 1092 leader_election.cc:302] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e, c3db1b5b143b4d5f831a65c805f446ee; no voters:
I20260808 08:22:42.620033 1394 raft_consensus.cc:2454] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d98465adfd4ba8a4449c6bf01f226e in term 1.
I20260808 08:22:42.620257 1708 raft_consensus.cc:2781] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:42.620584 1708 raft_consensus.cc:686] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 LEADER]: Becoming Leader. State: Replica: 45d98465adfd4ba8a4449c6bf01f226e, State: Running, Role: LEADER
I20260808 08:22:42.621006 1708 consensus_queue.cc:237] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:42.625167 957 catalog_manager.cc:5495] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1). New cstate: current_term: 1 leader_uuid: "45d98465adfd4ba8a4449c6bf01f226e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } health_report { overall_health: UNKNOWN } } }
W20260808 08:22:42.627452 1474 tablet.cc:2369] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:22:42.631851 1269 tablet.cc:2369] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:22:42.655019 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.650624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.12.13.1:44997)
0808 08:22:42.650786 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:42.650790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:42.650806 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:22:42.650945 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:42.650947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:42.651010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:42.651115 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:22:42.651459 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.652300 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.653321 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:42.653540 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:42.653628 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:42.654074 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:22:42.654536 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:22:42.654635 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:42.654751 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 08:22:42.654810 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:22:42.661399 1394 raft_consensus.cc:1261] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Refusing update from remote peer 45d98465adfd4ba8a4449c6bf01f226e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:42.661399 1606 raft_consensus.cc:1261] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Refusing update from remote peer 45d98465adfd4ba8a4449c6bf01f226e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:42.662220 1708 consensus_queue.cc:1035] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:42.662444 1714 consensus_queue.cc:1035] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:42.680761 955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:22:42.680971 955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:22:42.683607 955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Sending DeleteTablet for 3 replicas of tablet 2a4453a9cb04491fa92cee8ba2d35c6a
I20260808 08:22:42.684428 1167 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:42 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:38566
I20260808 08:22:42.684424 1570 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:42 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50798
I20260808 08:22:42.684566 1369 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:42 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:48610
I20260808 08:22:42.685021 1734 tablet_replica.cc:330] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: stopping tablet replica
I20260808 08:22:42.685130 1735 tablet_replica.cc:330] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: stopping tablet replica
I20260808 08:22:42.685207 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 1069
I20260808 08:22:42.685477 1735 raft_consensus.cc:2229] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:42.687803 1735 raft_consensus.cc:2258] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:42.693642 1733 tablet_replica.cc:330] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: stopping tablet replica
I20260808 08:22:42.694188 1733 raft_consensus.cc:2229] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:42.694676 1733 raft_consensus.cc:2258] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:42.698194 1733 ts_tablet_manager.cc:1905] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:22:42.703009 1733 ts_tablet_manager.cc:1918] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:22:42.703158 1733 log.cc:1196] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/2a4453a9cb04491fa92cee8ba2d35c6a
I20260808 08:22:42.703667 1733 ts_tablet_manager.cc:1939] T 2a4453a9cb04491fa92cee8ba2d35c6a P c3db1b5b143b4d5f831a65c805f446ee: Deleting consensus metadata
I20260808 08:22:42.705847 933 catalog_manager.cc:4841] TS c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): tablet 2a4453a9cb04491fa92cee8ba2d35c6a (table test-table [id=25433e3a41944862ba437addbd4f12e1]) successfully deleted
I20260808 08:22:42.711752 1735 ts_tablet_manager.cc:1905] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:22:42.714375 1735 ts_tablet_manager.cc:1918] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:22:42.714480 1735 log.cc:1196] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/2a4453a9cb04491fa92cee8ba2d35c6a
I20260808 08:22:42.714800 1735 ts_tablet_manager.cc:1939] T 2a4453a9cb04491fa92cee8ba2d35c6a P 7abf678e7bdb4c14a8e86982cf95a238: Deleting consensus metadata
I20260808 08:22:42.716171 929 catalog_manager.cc:4841] TS 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): tablet 2a4453a9cb04491fa92cee8ba2d35c6a (table test-table [id=25433e3a41944862ba437addbd4f12e1]) successfully deleted
W20260808 08:22:42.717612 934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:44997 (error 108)
I20260808 08:22:42.718343 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 1281
I20260808 08:22:42.722425 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:42.718506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:44997 (local address 127.0.0.1:38582)
0808 08:22:42.722142 (+ 3636us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:42.722252 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3499}
W20260808 08:22:42.724525 934 catalog_manager.cc:4584] TS 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a4453a9cb04491fa92cee8ba2d35c6a: Network error: Client connection negotiation failed: client connection to 127.12.13.1:44997: connect: Connection refused (error 111)
I20260808 08:22:42.734979 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 1479
I20260808 08:22:42.748724 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 904
I20260808 08:22:42.761053 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43219
--webserver_interface=127.12.13.62
--webserver_port=32869
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:42.881309 1738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:42.881623 1738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:42.881708 1738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:42.885753 1738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:42.885855 1738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:42.885885 1738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:42.885972 1738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:42.886008 1738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:42.891757 1738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:42.892964 1738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:42.894134 1738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:42.909785 1751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:42.909790 1752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:42.909785 1754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:42.910223 1738 server_base.cc:1034] running on GCE node
I20260808 08:22:42.910928 1738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:42.912222 1738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:42.913401 1738 hybrid_clock.cc:648] HybridClock initialized: now 1786177362913383 us; error 33 us; skew 500 ppm
Aug 08 08:22:42 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177362, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:42.916481 1738 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:42.917541 1738 webserver.cc:460] Webserver started at http://127.12.13.62:32869/ using document root <none> and password file <none>
I20260808 08:22:42.918272 1738 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:42.918382 1738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:42.922722 1738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 08:22:42.925182 1766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:42.926558 1738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:22:42.926697 1738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "84807906cc0a49eb97e47df9fed7b8e8"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "39eb98a459f972c185320a762cb9e6b9"
server_key_iv: "95f46c134344e59a65ff927dbf11decf"
server_key_version: "encryptionkey@0"
I20260808 08:22:42.927325 1738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:42.942013 1738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:42.945080 1738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:42.945302 1738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:42.953259 1738 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:43219
I20260808 08:22:42.953606 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:43219 every 8 connection(s)
I20260808 08:22:42.954578 1738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:42.958994 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 1738
I20260808 08:22:42.959507 1850 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:22:42.959528 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:44997
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:43219
--webserver_port=41833
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:42.962177 1850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap starting.
I20260808 08:22:42.980985 1850 log.cc:824] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:42.990345 1850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:22:42.990825 1850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap complete.
I20260808 08:22:42.995463 1850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:42.995864 1850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84807906cc0a49eb97e47df9fed7b8e8, State: Initialized, Role: FOLLOWER
I20260808 08:22:42.996390 1850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:42.996507 1850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:22:42.996603 1850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:22:42.996724 1850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 08:22:42.998790 1850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:42.999158 1850 leader_election.cc:302] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84807906cc0a49eb97e47df9fed7b8e8; no voters:
I20260808 08:22:42.999524 1850 leader_election.cc:288] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 08:22:42.999621 1855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 08:22:42.999840 1855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 LEADER]: Becoming Leader. State: Replica: 84807906cc0a49eb97e47df9fed7b8e8, State: Running, Role: LEADER
I20260808 08:22:43.000216 1855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:43.000871 1850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:43.001758 1858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84807906cc0a49eb97e47df9fed7b8e8. Latest consensus state: current_term: 2 leader_uuid: "84807906cc0a49eb97e47df9fed7b8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } } }
I20260808 08:22:43.001823 1857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84807906cc0a49eb97e47df9fed7b8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } } }
I20260808 08:22:43.001981 1857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:43.001921 1858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:43.002456 1866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:43.007784 1866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:43.008630 1866 catalog_manager.cc:1259] Loaded cluster ID: 8d3c6fadf65a40878fc39ba121af03ca
I20260808 08:22:43.008718 1866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:22:43.009939 1866 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 08:22:43.012759 1866 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 08:22:43.018057 1866 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:22:43.018803 1866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Loaded TSK: 0
I20260808 08:22:43.019829 1866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 08:22:43.095160 1853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:43.095451 1853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:43.095495 1853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:43.099779 1853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:43.099851 1853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:43.099912 1853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:22:43.105773 1853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:44997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=41833
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:43.107211 1853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:43.108325 1853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:43.118948 1886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:43.119175 1889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:43.118913 1887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:43.119495 1853 server_base.cc:1034] running on GCE node
I20260808 08:22:43.119942 1853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:43.120684 1853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:43.121950 1853 hybrid_clock.cc:648] HybridClock initialized: now 1786177363121914 us; error 44 us; skew 500 ppm
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177363, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:43.124955 1853 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:22:43.126129 1853 webserver.cc:460] Webserver started at http://127.12.13.1:41833/ using document root <none> and password file <none>
I20260808 08:22:43.126757 1853 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:43.126839 1853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:43.131191 1853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:22:43.133708 1900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.135146 1853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:22:43.135310 1853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "45d98465adfd4ba8a4449c6bf01f226e"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "3ea9704129ee6326f6889620eb3c3882"
server_key_iv: "b677ab56d2bed6772f1dc07ef6ba3c10"
server_key_version: "encryptionkey@0"
I20260808 08:22:43.135903 1853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:43.148146 1853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:43.151942 1853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:43.152166 1853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:43.152894 1853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:43.154220 1912 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 08:22:43.159927 1853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 08:22:43.160022 1853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 08:22:43.160130 1853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 08:22:43.162043 1853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 08:22:43.162113 1853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260808 08:22:43.162443 1912 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap starting.
I20260808 08:22:43.173298 1853 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:44997
I20260808 08:22:43.173372 2063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:44997 every 8 connection(s)
I20260808 08:22:43.174455 1853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:22:43.178251 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 1853
I20260808 08:22:43.178788 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:39597
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:43219
--webserver_port=44099
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:43.186545 1912 log.cc:824] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177363, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:43.194578 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.176658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:52161)
0808 08:22:43.177367 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.177425 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.178472 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.179841 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.179851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.180323 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:43.181043 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.181062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.182493 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.182496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.183378 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.183389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.184175 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.184179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.184383 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.185075 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:43.185097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:43.187221 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:43.190408 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.190418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.190436 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.190857 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.191371 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.191376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.191381 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.191593 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.192596 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:43.192604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:43.192925 (+ 321us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.193145 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.193498 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":145,"threads_started":1}
I20260808 08:22:43.195191 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.176752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:52161 (local address 127.12.13.62:43219)
0808 08:22:43.177179 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.177200 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.177568 (+ 368us) server_negotiation.cc:407] Connection header received
0808 08:22:43.179021 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.179039 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.179414 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.179764 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:43.181229 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.182309 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.183518 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.184009 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.184202 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.187479 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:43.187487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:43.187515 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:43.187563 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:43.190221 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.191022 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.191031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.191037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.191220 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.191726 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.191733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.191737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.192298 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:43.192551 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.193323 (+ 772us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.193706 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":149,"threads_started":1}
I20260808 08:22:43.195302 1912 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:22:43.195833 1912 tablet_bootstrap.cc:492] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap complete.
I20260808 08:22:43.197530 1912 ts_tablet_manager.cc:1397] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.004s
I20260808 08:22:43.198275 2064 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43219
I20260808 08:22:43.198541 2064 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:43.199113 2064 heartbeater.cc:511] Master 127.12.13.62:43219 requested a full tablet report, sending...
I20260808 08:22:43.200043 1912 raft_consensus.cc:348] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:43.200470 1912 raft_consensus.cc:729] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d98465adfd4ba8a4449c6bf01f226e, State: Initialized, Role: FOLLOWER
I20260808 08:22:43.201010 1912 consensus_queue.cc:260] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } }
I20260808 08:22:43.202111 1912 ts_tablet_manager.cc:1428] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260808 08:22:43.202118 1797 ts_manager.cc:194] Registered new tserver with Master: 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997)
I20260808 08:22:43.206372 1797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:52161
I20260808 08:22:43.209324 2064 heartbeater.cc:503] Master 127.12.13.62:43219 was elected leader, sending a full tablet report...
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177362, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.1@KRBTEST.COM
I20260808 08:22:43.216975 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.205720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.12.13.1:44997)
0808 08:22:43.206197 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.206209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.206618 (+ 409us) server_negotiation.cc:407] Connection header received
0808 08:22:43.206857 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.206869 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.207039 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.207187 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:43.208145 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.208907 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.210020 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.210382 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.210509 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.213146 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:43.213161 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:43.213171 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:43.213194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:43.214933 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.215447 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.215453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.215457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.215544 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.215843 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.215848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.215850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.216120 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:43.216283 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.216629 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.216802 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":379,"thread_start_us":139,"threads_started":1}
I20260808 08:22:43.216975 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.205667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:44997 (local address 127.0.0.1:38584)
0808 08:22:43.206458 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.206531 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.206692 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.207467 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.207472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.207904 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:43.208014 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.208021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.209036 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.209040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.209813 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.209826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.210512 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.210516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.210638 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.211356 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:43.211369 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:43.212998 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:43.215065 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.215075 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.215081 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.215324 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.215643 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.215647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.215649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.215750 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.216246 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:43.216252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:43.216395 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.216542 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.216782 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":115,"threads_started":1}
I20260808 08:22:43.218436 1979 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4453a9cb04491fa92cee8ba2d35c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:42 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:38584
I20260808 08:22:43.219537 2076 tablet_replica.cc:330] stopping tablet replica
I20260808 08:22:43.219892 2076 raft_consensus.cc:2229] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:43.220172 2076 raft_consensus.cc:2258] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:43.221256 2076 ts_tablet_manager.cc:1905] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:22:43.225210 2076 ts_tablet_manager.cc:1918] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:22:43.225340 2076 log.cc:1196] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/2a4453a9cb04491fa92cee8ba2d35c6a
I20260808 08:22:43.225626 2076 ts_tablet_manager.cc:1939] T 2a4453a9cb04491fa92cee8ba2d35c6a P 45d98465adfd4ba8a4449c6bf01f226e: Deleting consensus metadata
I20260808 08:22:43.227545 1775 catalog_manager.cc:4841] TS 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997): tablet 2a4453a9cb04491fa92cee8ba2d35c6a (table test-table [id=25433e3a41944862ba437addbd4f12e1]) successfully deleted
W20260808 08:22:43.312071 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:43.312367 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:43.312454 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:43.316727 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:43.316802 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:43.316896 2069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:22:43.322615 2069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:39597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=44099
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:43.323828 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:43.324976 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:22:43.335776 2069 server_base.cc:1034] running on GCE node
W20260808 08:22:43.335747 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:43.335711 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:43.335733 2082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:43.336472 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:43.337172 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:43.338351 2069 hybrid_clock.cc:648] HybridClock initialized: now 1786177363338313 us; error 49 us; skew 500 ppm
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177363, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:43.341588 2069 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:22:43.342602 2069 webserver.cc:460] Webserver started at http://127.12.13.2:44099/ using document root <none> and password file <none>
I20260808 08:22:43.343305 2069 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:43.343442 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:43.347697 2069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 08:22:43.350060 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.351470 2069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:22:43.351625 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "7abf678e7bdb4c14a8e86982cf95a238"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "637584bcd970d14266cb6b5f3fe51df9"
server_key_iv: "da05daad97a0dfafc5c0a7b6bcba715d"
server_key_version: "encryptionkey@0"
I20260808 08:22:43.352294 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:43.381537 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:43.385010 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:43.385242 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:43.385959 2069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:43.386874 2069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:43.386948 2069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.387022 2069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:43.387073 2069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.396903 2069 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:39597
I20260808 08:22:43.396924 2265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:39597 every 8 connection(s)
I20260808 08:22:43.398037 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:22:43.401337 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 2069
I20260808 08:22:43.401949 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:36769
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:43219
--webserver_port=41661
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177363, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:43.414099 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.399905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:55381 (local address 127.12.13.62:43219)
0808 08:22:43.400053 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.400057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.400913 (+ 856us) server_negotiation.cc:407] Connection header received
0808 08:22:43.403091 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.403094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.403149 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.403227 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:43.404744 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.405273 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.406572 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.406801 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.406927 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.409960 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:43.409963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:43.409965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:43.409994 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:43.411549 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.412288 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.412291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.412292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.412372 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.412810 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.412812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.412813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.413001 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:43.413074 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.413781 (+ 707us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.413969 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:22:43.415412 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.400254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:55381)
0808 08:22:43.400737 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.400780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.402806 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.403478 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.403490 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.404046 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:43.404574 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.404588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.405439 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.405443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.406427 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.406437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.406970 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.406974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.407140 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.407764 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:43.407781 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:43.409699 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:43.411676 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.411685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.411702 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.412181 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.412476 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.412481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.412485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.412697 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.413116 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:43.413126 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:43.413498 (+ 372us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.413758 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.414204 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260808 08:22:43.416689 2266 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43219
I20260808 08:22:43.416968 2266 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:43.417498 2266 heartbeater.cc:511] Master 127.12.13.62:43219 requested a full tablet report, sending...
I20260808 08:22:43.418679 1797 ts_manager.cc:194] Registered new tserver with Master: 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
I20260808 08:22:43.419262 1797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:55381
W20260808 08:22:43.526479 2269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:43.526829 2269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:43.526904 2269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:43.531220 2269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:43.531279 2269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:43.531385 2269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:22:43.536868 2269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=41661
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:43.538316 2269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:43.539649 2269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:43.550246 2284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:43.550457 2269 server_base.cc:1034] running on GCE node
W20260808 08:22:43.550228 2282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:43.550228 2281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:43.551306 2269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:43.552037 2269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:43.553253 2269 hybrid_clock.cc:648] HybridClock initialized: now 1786177363553228 us; error 37 us; skew 500 ppm
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177363, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:43.556224 2269 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:22:43.557183 2269 webserver.cc:460] Webserver started at http://127.12.13.3:41661/ using document root <none> and password file <none>
I20260808 08:22:43.557816 2269 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:43.557971 2269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:43.562101 2269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 08:22:43.564172 2298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.565420 2269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:22:43.565521 2269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "c3db1b5b143b4d5f831a65c805f446ee"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "2a62672e15e73e6c6d72b40d0eec0ea8"
server_key_iv: "fd30229ef665ed11b641890e8ceea788"
server_key_version: "encryptionkey@0"
I20260808 08:22:43.566140 2269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:43.588732 2269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:43.592520 2269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:43.592687 2269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:43.593354 2269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:43.594292 2269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:43.594360 2269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.594449 2269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:43.594489 2269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:43.605322 2269 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:36769
I20260808 08:22:43.605382 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:36769 every 8 connection(s)
I20260808 08:22:43.606493 2269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:22:43.610574 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 2269
Aug 08 08:22:43 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177363, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:22:43.620066 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.611270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.12.13.62:43219)
0808 08:22:43.611542 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.611546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.611687 (+ 141us) server_negotiation.cc:407] Connection header received
0808 08:22:43.611793 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.611796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.611841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.611935 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:43.612654 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.613221 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.614195 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.614553 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.614642 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.616113 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:43.616118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:43.616121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:43.616146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:43.618009 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.618518 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.618521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.618522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.618618 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.619034 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.619047 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.619050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.619265 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:43.619428 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.619690 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.619855 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"spinlock_wait_cycles":4992,"thread_start_us":104,"threads_started":1}
I20260808 08:22:43.622159 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.608390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:51993 (local address 127.12.13.62:43219)
0808 08:22:43.608573 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.608578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.609466 (+ 888us) server_negotiation.cc:407] Connection header received
0808 08:22:43.611050 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.611054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.611130 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.611232 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:43.612735 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.613226 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.614563 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.614894 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.615011 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.617958 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:43.617961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:43.617962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:43.617997 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:43.619708 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.620337 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.620340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.620342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.620422 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:43.620805 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:43.620807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:43.620809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:43.621008 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:43.621112 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.621814 (+ 702us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.621981 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"spinlock_wait_cycles":768}
I20260808 08:22:43.623102 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.608820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:51993)
0808 08:22:43.609302 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.609345 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.610797 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.611459 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.611468 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.612046 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:43.612561 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.612575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.613363 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.613367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.614437 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.614448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.615078 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.615083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.615317 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.615905 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:43.615922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:43.617724 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:43.619830 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.619836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.619854 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.620219 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.620524 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:43.620527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:43.620529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:43.620693 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:43.621160 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:43.621168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:43.621519 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.621702 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.622102 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260808 08:22:43.624449 2469 heartbeater.cc:347] Connected to a master server at 127.12.13.62:43219
I20260808 08:22:43.624742 2469 heartbeater.cc:464] Registering TS with master...
I20260808 08:22:43.625310 2469 heartbeater.cc:511] Master 127.12.13.62:43219 requested a full tablet report, sending...
I20260808 08:22:43.626523 1797 ts_manager.cc:194] Registered new tserver with Master: c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769)
I20260808 08:22:43.627067 1797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:51993
I20260808 08:22:43.634716 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:22:43.639639 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.635385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.12.13.62:43219)
0808 08:22:43.635553 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.635559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.635743 (+ 184us) server_negotiation.cc:407] Connection header received
0808 08:22:43.635972 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.635975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.636087 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.636205 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:22:43.636478 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.636996 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.637987 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.638215 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.638337 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.638728 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:22:43.639193 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:22:43.639371 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.639405 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.639483 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:22:43.640389 1797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:22:43.643942 1797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:22:43.652724 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6c6f84c0d5414e9111f1eeedccac8c (DEFAULT_TABLE table=test-table [id=9aa48e50386049d2816ebc72ec9d041f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:43.653486 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6c6f84c0d5414e9111f1eeedccac8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:43.656826 1912 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap starting.
I20260808 08:22:43.657122 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.651415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:39597 (local address 127.0.0.1:48624)
0808 08:22:43.651678 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.651688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.651755 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.652554 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.652557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.652576 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:43.652692 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.652701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.654076 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.654081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.655080 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.655091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.656595 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.656599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.656694 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.656754 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.656836 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.656933 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":135,"threads_started":1}
I20260808 08:22:43.657330 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.651130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36769 (local address 127.0.0.1:50804)
0808 08:22:43.651330 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.651370 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.651449 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.652711 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.652714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.652734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:43.652846 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.652853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.654679 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.654683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.655592 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.655600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.656877 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.656881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.656983 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.657007 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.657100 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.657188 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 08:22:43.657505 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.651446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.12.13.3:36769)
0808 08:22:43.652211 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.652225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.652252 (+ 27us) server_negotiation.cc:407] Connection header received
0808 08:22:43.652336 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.652341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.652542 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.653168 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:43.653336 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.654529 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.655732 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.656700 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.656848 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.656984 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.657133 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.657301 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":125,"threads_started":1}
I20260808 08:22:43.657799 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.651715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.12.13.2:39597)
0808 08:22:43.652078 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.652092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.652115 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:22:43.652200 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.652206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.652406 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.652578 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:43.652799 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.653924 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.655231 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.656307 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.656596 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.656772 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.657513 (+ 741us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.657616 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260808 08:22:43.657895 1912 tablet_bootstrap.cc:654] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:43.660073 1912 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: No bootstrap required, opened a new log
I20260808 08:22:43.660175 1912 ts_tablet_manager.cc:1397] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:22:43.660670 1912 raft_consensus.cc:348] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.660789 1912 raft_consensus.cc:374] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:43.660826 1912 raft_consensus.cc:729] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d98465adfd4ba8a4449c6bf01f226e, State: Initialized, Role: FOLLOWER
I20260808 08:22:43.660964 1912 consensus_queue.cc:260] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.660719 2377 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6c6f84c0d5414e9111f1eeedccac8c (DEFAULT_TABLE table=test-table [id=9aa48e50386049d2816ebc72ec9d041f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:43.661912 2377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6c6f84c0d5414e9111f1eeedccac8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:43.661952 1912 ts_tablet_manager.cc:1428] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 08:22:43.661805 2169 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6c6f84c0d5414e9111f1eeedccac8c (DEFAULT_TABLE table=test-table [id=9aa48e50386049d2816ebc72ec9d041f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:22:43.662967 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6c6f84c0d5414e9111f1eeedccac8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:22:43.665715 2071 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:43.665939 2071 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.667179 2071 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
I20260808 08:22:43.672081 2491 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Bootstrap starting.
I20260808 08:22:43.672698 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.667506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:39597 (local address 127.12.13.1:46899)
0808 08:22:43.668130 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.668145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.668249 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.668507 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.668510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.668530 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:43.668696 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.668705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.669696 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.669699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.671163 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.671178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.672139 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.672144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.672260 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.672323 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.672421 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.672499 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260808 08:22:43.673055 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.667864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:46899 (local address 127.12.13.2:39597)
0808 08:22:43.668132 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.668138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.668232 (+ 94us) server_negotiation.cc:407] Connection header received
0808 08:22:43.668335 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.668338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.668391 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.668476 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:43.668811 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.669558 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.671310 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.671879 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.671985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.672050 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.672876 (+ 826us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.672950 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260808 08:22:43.673679 2490 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Bootstrap starting.
I20260808 08:22:43.673830 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.667365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36769 (local address 127.12.13.1:59723)
0808 08:22:43.667628 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.667646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.667743 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.670146 (+ 2403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.670149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.670166 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:43.670401 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.670409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.671571 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.671576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.672622 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.672632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.673454 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.673459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.673560 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.673587 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.673641 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.673713 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":96}
I20260808 08:22:43.674497 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.667927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:59723 (local address 127.12.13.3:36769)
0808 08:22:43.669940 (+ 2013us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.669944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.669965 (+ 21us) server_negotiation.cc:407] Connection header received
0808 08:22:43.670003 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.670005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.670040 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.670133 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:43.670534 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.671211 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.672758 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.673304 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.674189 (+ 885us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.674265 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.674306 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.674376 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1923}
I20260808 08:22:43.674981 2491 tablet_bootstrap.cc:654] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Neither blocks nor log segments found. Creating new log.
I20260808 08:22:43.674278 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6c6f84c0d5414e9111f1eeedccac8c" candidate_uuid: "45d98465adfd4ba8a4449c6bf01f226e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf678e7bdb4c14a8e86982cf95a238" is_pre_election: true
W20260808 08:22:43.675743 2066 tablet.cc:2369] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:22:43.675933 2491 log.cc:824] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:43.676069 2490 tablet_bootstrap.cc:654] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Neither blocks nor log segments found. Creating new log.
W20260808 08:22:43.676280 1901 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): Illegal state: must be running to vote when last-logged opid is not known
I20260808 08:22:43.677477 2490 log.cc:824] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:43.677076 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6c6f84c0d5414e9111f1eeedccac8c" candidate_uuid: "45d98465adfd4ba8a4449c6bf01f226e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3db1b5b143b4d5f831a65c805f446ee" is_pre_election: true
I20260808 08:22:43.678639 2491 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: No bootstrap required, opened a new log
I20260808 08:22:43.678865 2491 ts_tablet_manager.cc:1397] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
W20260808 08:22:43.679029 1903 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): Illegal state: must be running to vote when last-logged opid is not known
I20260808 08:22:43.679139 1903 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e; no voters: 7abf678e7bdb4c14a8e86982cf95a238, c3db1b5b143b4d5f831a65c805f446ee
I20260808 08:22:43.679668 2071 raft_consensus.cc:2726] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 08:22:43.679787 2490 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: No bootstrap required, opened a new log
I20260808 08:22:43.680004 2490 ts_tablet_manager.cc:1397] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:22:43.681924 2491 raft_consensus.cc:348] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.682212 2491 raft_consensus.cc:374] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:43.682268 2491 raft_consensus.cc:729] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abf678e7bdb4c14a8e86982cf95a238, State: Initialized, Role: FOLLOWER
I20260808 08:22:43.682763 2491 consensus_queue.cc:260] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.682878 2490 raft_consensus.cc:348] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.683122 2490 raft_consensus.cc:374] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:22:43.683184 2490 raft_consensus.cc:729] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3db1b5b143b4d5f831a65c805f446ee, State: Initialized, Role: FOLLOWER
I20260808 08:22:43.683526 2266 heartbeater.cc:503] Master 127.12.13.62:43219 was elected leader, sending a full tablet report...
I20260808 08:22:43.683653 2490 consensus_queue.cc:260] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.683962 2491 ts_tablet_manager.cc:1428] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:22:43.684355 2469 heartbeater.cc:503] Master 127.12.13.62:43219 was elected leader, sending a full tablet report...
I20260808 08:22:43.684636 2490 ts_tablet_manager.cc:1428] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:22:43.774718 2497 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:43.774999 2497 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.776253 2497 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597), 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997)
I20260808 08:22:43.780391 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.776564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:44997 (local address 127.12.13.3:51601)
0808 08:22:43.776808 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.776826 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.776930 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.777255 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.777258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.777276 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:43.777420 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.777426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.778229 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.778232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.779178 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.779189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.779961 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.779965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.780064 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.780086 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.780155 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.780231 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":21}
I20260808 08:22:43.780678 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.776707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:51601 (local address 127.12.13.1:44997)
0808 08:22:43.777016 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.777022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.777038 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:22:43.777096 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.777098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.777137 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.777241 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:43.777542 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.778095 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.779300 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.779825 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.779958 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.780123 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.780423 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.780520 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260808 08:22:43.781163 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.776813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:53815 (local address 127.12.13.2:39597)
0808 08:22:43.776987 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.776992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.777185 (+ 193us) server_negotiation.cc:407] Connection header received
0808 08:22:43.777373 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.777379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.777445 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.777546 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:22:43.777926 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.778511 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.779813 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.780369 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.780505 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.780596 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.780974 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.781051 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 08:22:43.781153 2008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6c6f84c0d5414e9111f1eeedccac8c" candidate_uuid: "c3db1b5b143b4d5f831a65c805f446ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d98465adfd4ba8a4449c6bf01f226e" is_pre_election: true
I20260808 08:22:43.781433 2008 raft_consensus.cc:2454] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3db1b5b143b4d5f831a65c805f446ee in term 0.
I20260808 08:22:43.781543 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.776626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:39597 (local address 127.12.13.3:53815)
0808 08:22:43.777073 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.777086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:43.777232 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:43.777564 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:43.777567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:43.777587 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:22:43.777739 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.777745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.778632 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.778638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.779675 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:43.779686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.780706 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.780710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:43.780820 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.780843 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:22:43.781304 (+ 461us) client_negotiation.cc:240] Negotiation successful
0808 08:22:43.781391 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":1,"thread_start_us":158,"threads_started":1}
I20260808 08:22:43.781988 2302 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e, c3db1b5b143b4d5f831a65c805f446ee; no voters:
I20260808 08:22:43.782066 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6c6f84c0d5414e9111f1eeedccac8c" candidate_uuid: "c3db1b5b143b4d5f831a65c805f446ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf678e7bdb4c14a8e86982cf95a238" is_pre_election: true
I20260808 08:22:43.782310 2497 raft_consensus.cc:2781] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:22:43.782385 2206 raft_consensus.cc:2454] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3db1b5b143b4d5f831a65c805f446ee in term 0.
I20260808 08:22:43.782423 2497 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:22:43.782471 2497 raft_consensus.cc:3037] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:43.783577 2497 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.783907 2497 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [CANDIDATE]: Term 1 election: Requested vote from peers 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597), 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997)
I20260808 08:22:43.784289 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6c6f84c0d5414e9111f1eeedccac8c" candidate_uuid: "c3db1b5b143b4d5f831a65c805f446ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf678e7bdb4c14a8e86982cf95a238"
I20260808 08:22:43.784289 2008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6c6f84c0d5414e9111f1eeedccac8c" candidate_uuid: "c3db1b5b143b4d5f831a65c805f446ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d98465adfd4ba8a4449c6bf01f226e"
I20260808 08:22:43.784441 2008 raft_consensus.cc:3037] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:43.784442 2206 raft_consensus.cc:3037] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:22:43.785926 2008 raft_consensus.cc:2454] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3db1b5b143b4d5f831a65c805f446ee in term 1.
I20260808 08:22:43.785928 2206 raft_consensus.cc:2454] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3db1b5b143b4d5f831a65c805f446ee in term 1.
I20260808 08:22:43.786247 2302 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e, c3db1b5b143b4d5f831a65c805f446ee; no voters:
I20260808 08:22:43.786509 2497 raft_consensus.cc:2781] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:22:43.786777 2497 raft_consensus.cc:686] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 1 LEADER]: Becoming Leader. State: Replica: c3db1b5b143b4d5f831a65c805f446ee, State: Running, Role: LEADER
I20260808 08:22:43.787179 2497 consensus_queue.cc:237] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:43.790879 1797 catalog_manager.cc:5495] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee reported cstate change: term changed from 0 to 1, leader changed from <none> to c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3). New cstate: current_term: 1 leader_uuid: "c3db1b5b143b4d5f831a65c805f446ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } health_report { overall_health: UNKNOWN } } }
I20260808 08:22:43.820111 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.816627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.12.13.3:36769)
0808 08:22:43.816815 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:43.816819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:43.816833 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:22:43.816939 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:43.816941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:43.817002 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:43.817104 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:22:43.817348 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.817862 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.818718 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:43.818942 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:43.819018 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:43.819302 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:22:43.819755 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:22:43.819850 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:43.819940 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 08:22:43.819989 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 08:22:43.826860 2206 raft_consensus.cc:1261] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Refusing update from remote peer c3db1b5b143b4d5f831a65c805f446ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:43.826894 2008 raft_consensus.cc:1261] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Refusing update from remote peer c3db1b5b143b4d5f831a65c805f446ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:22:43.827656 2502 consensus_queue.cc:1035] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:43.827881 2497 consensus_queue.cc:1035] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:22:43.835201 2508 mvcc.cc:205] Tried to move back new op lower bound from 7316182482225991680 to 7316182482075049984. Current Snapshot: MvccSnapshot[applied={T|T < 7316182482225991680}]
I20260808 08:22:43.844503 1797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:22:43.844707 1797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:22:43.847373 1797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Sending DeleteTablet for 3 replicas of tablet 4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:43.848127 1979 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6c6f84c0d5414e9111f1eeedccac8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:43 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:38584
I20260808 08:22:43.848202 2377 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6c6f84c0d5414e9111f1eeedccac8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:43 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50804
I20260808 08:22:43.848210 2169 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6c6f84c0d5414e9111f1eeedccac8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:22:43 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:48624
I20260808 08:22:43.848733 2519 tablet_replica.cc:330] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: stopping tablet replica
I20260808 08:22:43.848937 2520 tablet_replica.cc:330] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: stopping tablet replica
I20260808 08:22:43.848953 2519 raft_consensus.cc:2229] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:43.849191 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 1853
I20260808 08:22:43.849210 2519 raft_consensus.cc:2258] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:43.849278 2520 raft_consensus.cc:2229] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:22:43.859962 2520 raft_consensus.cc:2258] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:43.862339 2520 ts_tablet_manager.cc:1905] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:22:43.866174 2522 tablet_replica.cc:330] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: stopping tablet replica
I20260808 08:22:43.866619 2522 raft_consensus.cc:2229] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:22:43.866863 2520 ts_tablet_manager.cc:1918] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:22:43.867031 2522 raft_consensus.cc:2258] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:22:43.867060 2520 log.cc:1196] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:43.867633 2520 ts_tablet_manager.cc:1939] T 4b6c6f84c0d5414e9111f1eeedccac8c P c3db1b5b143b4d5f831a65c805f446ee: Deleting consensus metadata
I20260808 08:22:43.868765 2522 ts_tablet_manager.cc:1905] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:22:43.869274 1771 catalog_manager.cc:4841] TS c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): tablet 4b6c6f84c0d5414e9111f1eeedccac8c (table test-table [id=9aa48e50386049d2816ebc72ec9d041f]) successfully deleted
I20260808 08:22:43.871624 2522 ts_tablet_manager.cc:1918] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:22:43.871753 2522 log.cc:1196] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:43.872099 2522 ts_tablet_manager.cc:1939] T 4b6c6f84c0d5414e9111f1eeedccac8c P 7abf678e7bdb4c14a8e86982cf95a238: Deleting consensus metadata
W20260808 08:22:43.872709 1775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:44997 (error 108)
I20260808 08:22:43.873425 1768 catalog_manager.cc:4841] TS 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): tablet 4b6c6f84c0d5414e9111f1eeedccac8c (table test-table [id=9aa48e50386049d2816ebc72ec9d041f]) successfully deleted
I20260808 08:22:43.873615 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 2069
I20260808 08:22:43.876629 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:43.873805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:44997 (local address 127.0.0.1:38594)
0808 08:22:43.876447 (+ 2642us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:43.876533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2535}
W20260808 08:22:43.878974 1775 catalog_manager.cc:4584] TS 45d98465adfd4ba8a4449c6bf01f226e (127.12.13.1:44997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b6c6f84c0d5414e9111f1eeedccac8c: Network error: Client connection negotiation failed: client connection to 127.12.13.1:44997: connect: Connection refused (error 111)
I20260808 08:22:43.887336 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 2269
I20260808 08:22:43.906934 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 1738
I20260808 08:22:43.921149 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43219
--webserver_interface=127.12.13.62
--webserver_port=32869
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:22:44.040228 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:44.040529 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:44.040570 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:44.044732 2525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:22:44.044806 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:44.044826 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:44.044843 2525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:22:44.044858 2525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:22:44.050616 2525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:44.051792 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:44.052913 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:44.062695 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.062717 2543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.062705 2536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:44.063004 2525 server_base.cc:1034] running on GCE node
I20260808 08:22:44.063789 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:44.065111 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:44.066321 2525 hybrid_clock.cc:648] HybridClock initialized: now 1786177364066304 us; error 28 us; skew 500 ppm
Aug 08 08:22:44 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:44.069437 2525 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:22:44.070518 2525 webserver.cc:460] Webserver started at http://127.12.13.62:32869/ using document root <none> and password file <none>
I20260808 08:22:44.071130 2525 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:44.071231 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:44.075342 2525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:22:44.077548 2559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.079108 2525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:22:44.079325 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "84807906cc0a49eb97e47df9fed7b8e8"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "39eb98a459f972c185320a762cb9e6b9"
server_key_iv: "95f46c134344e59a65ff927dbf11decf"
server_key_version: "encryptionkey@0"
I20260808 08:22:44.079953 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:44.105826 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:44.109843 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:44.110234 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:44.119753 2525 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:43219
I20260808 08:22:44.120078 2636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:43219 every 8 connection(s)
I20260808 08:22:44.121320 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:22:44.126397 2637 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:22:44.128335 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap starting.
I20260808 08:22:44.129415 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 2525
I20260808 08:22:44.129972 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:44997
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:43219
--webserver_port=41833
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:44.147590 2637 log.cc:824] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:44.159214 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:22:44.159629 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8: Bootstrap complete.
I20260808 08:22:44.163967 2637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:44.164403 2637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84807906cc0a49eb97e47df9fed7b8e8, State: Initialized, Role: FOLLOWER
I20260808 08:22:44.164974 2637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:44.165125 2637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:22:44.165220 2637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:22:44.165337 2637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 08:22:44.167412 2637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:44.167773 2637 leader_election.cc:302] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84807906cc0a49eb97e47df9fed7b8e8; no voters:
I20260808 08:22:44.168126 2637 leader_election.cc:288] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 08:22:44.168223 2642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 08:22:44.168462 2642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [term 3 LEADER]: Becoming Leader. State: Replica: 84807906cc0a49eb97e47df9fed7b8e8, State: Running, Role: LEADER
I20260808 08:22:44.168772 2642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } }
I20260808 08:22:44.169485 2637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:22:44.169819 2644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84807906cc0a49eb97e47df9fed7b8e8. Latest consensus state: current_term: 3 leader_uuid: "84807906cc0a49eb97e47df9fed7b8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } } }
I20260808 08:22:44.169974 2644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:44.170236 2643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "84807906cc0a49eb97e47df9fed7b8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84807906cc0a49eb97e47df9fed7b8e8" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 43219 } } }
I20260808 08:22:44.170346 2643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84807906cc0a49eb97e47df9fed7b8e8 [sys.catalog]: This master's current role is: LEADER
I20260808 08:22:44.170385 2646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:22:44.175818 2646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:22:44.176620 2646 catalog_manager.cc:1259] Loaded cluster ID: 8d3c6fadf65a40878fc39ba121af03ca
I20260808 08:22:44.176684 2646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 08:22:44.177834 2646 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc07becc9ad google::LogMessage::Fail() at ??:0
@ 0x7fc07becf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc07becc4de google::LogMessage::Flush() at ??:0
@ 0x7fc07becda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc08680ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc086810d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc08680d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc086851640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc07e11e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc07e11ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc07e1203f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc07e10f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc07f7536db start_thread at ??:0
@ 0x7fc07a9ca61f clone at ??:0
W20260808 08:22:44.275925 2639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:44.276252 2639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:44.276306 2639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:44.281322 2639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:44.281410 2639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:44.281507 2639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:22:44.287503 2639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:44997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=41833
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:44.288730 2639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:44.289839 2639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:44.300483 2705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.300719 2706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.300505 2708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:44.300772 2639 server_base.cc:1034] running on GCE node
I20260808 08:22:44.301348 2639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:44.302147 2639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:44.303352 2639 hybrid_clock.cc:648] HybridClock initialized: now 1786177364303322 us; error 49 us; skew 500 ppm
Aug 08 08:22:44 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:44.306568 2639 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:22:44.307659 2639 webserver.cc:460] Webserver started at http://127.12.13.1:41833/ using document root <none> and password file <none>
I20260808 08:22:44.308369 2639 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:44.308483 2639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:44.313421 2639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:22:44.316195 2717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.317754 2639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 08:22:44.317941 2639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "45d98465adfd4ba8a4449c6bf01f226e"
format_stamp: "Formatted at 2026-08-08 08:22:41 on dist-test-slave-6ntk"
server_key: "3ea9704129ee6326f6889620eb3c3882"
server_key_iv: "b677ab56d2bed6772f1dc07ef6ba3c10"
server_key_version: "encryptionkey@0"
I20260808 08:22:44.318621 2639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:44.348104 2639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:44.351853 2639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:44.352100 2639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:44.352799 2639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:44.354137 2726 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 08:22:44.360946 2639 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 08:22:44.361074 2639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260808 08:22:44.361172 2639 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 08:22:44.363059 2639 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 08:22:44.363137 2639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260808 08:22:44.363473 2726 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap starting.
I20260808 08:22:44.374830 2639 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:44997
I20260808 08:22:44.375146 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:44997 every 8 connection(s)
I20260808 08:22:44.375967 2639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:22:44.383287 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 2639
I20260808 08:22:44.383749 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:39597
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:43219
--webserver_port=44099
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:44.393687 2726 log.cc:824] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Log is configured to *not* fsync() on all Append() calls
I20260808 08:22:44.403606 2726 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:22:44.404032 2726 tablet_bootstrap.cc:492] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Bootstrap complete.
I20260808 08:22:44.405346 2726 ts_tablet_manager.cc:1397] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.012s
I20260808 08:22:44.408056 2726 raft_consensus.cc:348] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:44.408525 2726 raft_consensus.cc:729] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d98465adfd4ba8a4449c6bf01f226e, State: Initialized, Role: FOLLOWER
I20260808 08:22:44.409050 2726 consensus_queue.cc:260] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:44.410171 2726 ts_tablet_manager.cc:1428] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260808 08:22:44.484061 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:22:44.378162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:53761)
0808 08:22:44.378762 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.378820 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:44.379763 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:44.483209 (+103446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":175,"threads_started":1}
W20260808 08:22:44.484799 2876 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:43219 (0 consecutive failures): Network error: Failed to ping master at 127.12.13.62:43219: Client connection negotiation failed: client connection to 127.12.13.62:43219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 08:22:44.485733 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.485404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:45423)
0808 08:22:44.485566 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.485650 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 08:22:44.486546 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.486297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:52419)
0808 08:22:44.486430 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.486484 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 08:22:44.486757 2876 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:22:44.529464 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:44.529754 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:44.529801 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:44.534034 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:44.534099 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:44.534193 2880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:22:44.539669 2880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:39597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=44099
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:44.540861 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:44.542076 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:44.552616 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.552641 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.552641 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:44.552853 2880 server_base.cc:1034] running on GCE node
I20260808 08:22:44.553501 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:44.554301 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:44.555539 2880 hybrid_clock.cc:648] HybridClock initialized: now 1786177364555495 us; error 58 us; skew 500 ppm
Aug 08 08:22:44 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:44.558708 2880 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:22:44.559705 2880 webserver.cc:460] Webserver started at http://127.12.13.2:44099/ using document root <none> and password file <none>
I20260808 08:22:44.560410 2880 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:44.560518 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:44.564808 2880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:22:44.567241 2907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.568657 2880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:22:44.568833 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "7abf678e7bdb4c14a8e86982cf95a238"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "637584bcd970d14266cb6b5f3fe51df9"
server_key_iv: "da05daad97a0dfafc5c0a7b6bcba715d"
server_key_version: "encryptionkey@0"
I20260808 08:22:44.569505 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:44.588325 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:44.592199 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:44.592451 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:44.593168 2880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:44.594187 2880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:44.594269 2880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.594357 2880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:44.594408 2880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.605482 2880 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:39597
I20260808 08:22:44.605513 3074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:39597 every 8 connection(s)
I20260808 08:22:44.606680 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:22:44.607707 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 2880
I20260808 08:22:44.608347 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:36769
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:43219
--webserver_port=41661
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:22:44.611856 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.609364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:56395)
0808 08:22:44.609917 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.610057 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":148,"threads_started":1}
W20260808 08:22:44.612389 3077 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:43219 (0 consecutive failures): Network error: Failed to ping master at 127.12.13.62:43219: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
I20260808 08:22:44.613102 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.612864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:50219)
0808 08:22:44.612986 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.613039 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 08:22:44.614002 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.613625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:49029)
0808 08:22:44.613799 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.613905 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 08:22:44.614300 3077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:22:44.733731 3080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:22:44.734086 3080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:22:44.734169 3080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:22:44.738365 3080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:22:44.738483 3080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:22:44.738581 3080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:22:44.744000 3080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=41661
--tserver_master_addrs=127.12.13.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:22:44.745162 3080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:22:44.746312 3080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:22:44.756837 3088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.756914 3090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:22:44.756880 3095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:22:44.757059 3080 server_base.cc:1034] running on GCE node
I20260808 08:22:44.757723 3080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:22:44.758505 3080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:22:44.759683 3080 hybrid_clock.cc:648] HybridClock initialized: now 1786177364759661 us; error 34 us; skew 500 ppm
Aug 08 08:22:44 dist-test-slave-6ntk krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:22:44.762666 3080 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:22:44.763700 3080 webserver.cc:460] Webserver started at http://127.12.13.3:41661/ using document root <none> and password file <none>
I20260808 08:22:44.764346 3080 fs_manager.cc:359] Metadata directory not provided
I20260808 08:22:44.764449 3080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:22:44.768749 3080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:22:44.771145 3105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.772526 3080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:22:44.772675 3080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "c3db1b5b143b4d5f831a65c805f446ee"
format_stamp: "Formatted at 2026-08-08 08:22:42 on dist-test-slave-6ntk"
server_key: "2a62672e15e73e6c6d72b40d0eec0ea8"
server_key_iv: "fd30229ef665ed11b641890e8ceea788"
server_key_version: "encryptionkey@0"
I20260808 08:22:44.773334 3080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:22:44.789162 3080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:22:44.792403 3080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:22:44.792622 3080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:22:44.793278 3080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:22:44.794191 3080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:22:44.794267 3080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.794344 3080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:22:44.794389 3080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:22:44.804987 3080 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:36769
I20260808 08:22:44.805014 3265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:36769 every 8 connection(s)
I20260808 08:22:44.806177 3080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:22:44.808362 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 3080
I20260808 08:22:44.810532 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.808435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:46095)
0808 08:22:44.808977 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.809095 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":118,"threads_started":1}
W20260808 08:22:44.811060 3267 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:43219 (0 consecutive failures): Network error: Failed to ping master at 127.12.13.62:43219: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
I20260808 08:22:44.811663 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.811455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:39633)
0808 08:22:44.811562 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.811609 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 08:22:44.812279 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:44.812084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:37781)
0808 08:22:44.812180 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:44.812230 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 08:22:44.812500 3267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 08:22:45.487898 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.487421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:50557)
0808 08:22:45.487699 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:45.487761 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260808 08:22:45.615355 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.614914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:53253)
0808 08:22:45.615180 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:45.615239 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260808 08:22:45.813448 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.813032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:50965)
0808 08:22:45.813283 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:45.813341 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260808 08:22:45.946358 3277 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:45.946561 3277 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:45.947618 3277 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
Aug 08 08:22:45 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.2@KRBTEST.COM
Aug 08 08:22:45 dist-test-slave-6ntk krb5kdc[889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.3@KRBTEST.COM
I20260808 08:22:45.961735 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.947886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:39597 (local address 127.12.13.1:56925)
0808 08:22:45.948007 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:45.948023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:45.948140 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:45.950142 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:45.950145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:45.950707 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:45.951206 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:45.951222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.952620 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.952625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:45.953639 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:45.953649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.954560 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.954564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:45.954811 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:45.955530 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:45.955542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:45.957157 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:45.959372 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:45.959381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:45.959394 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:45.959650 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:45.960006 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:45.960008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:45.960010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:45.960151 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:45.960794 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:45.960800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:45.961026 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 08:22:45.961213 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 08:22:45.961542 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":106,"spinlock_wait_cycles":15104}
I20260808 08:22:45.961812 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.948140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56925 (local address 127.12.13.2:39597)
0808 08:22:45.948571 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:45.948603 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:45.948652 (+ 49us) server_negotiation.cc:407] Connection header received
0808 08:22:45.949385 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:45.949391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:45.949734 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:45.950099 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:45.951467 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.952473 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.953920 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.954454 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.954623 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:45.957392 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:45.957397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:45.957409 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:45.957437 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:45.959249 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:45.959770 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:45.959776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:45.959789 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:45.959898 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:45.960248 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:45.960253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:45.960255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:45.960664 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:45.960807 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:45.961326 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 08:22:45.961643 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":147,"threads_started":1}
I20260808 08:22:45.961912 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.948107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:57317 (local address 127.12.13.3:36769)
0808 08:22:45.948621 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0808 08:22:45.948642 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:22:45.948678 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:22:45.949385 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:22:45.949391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:22:45.949746 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:22:45.950098 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:22:45.951515 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.952516 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.953772 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.954471 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.954634 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:45.957725 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:22:45.957732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:22:45.957746 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:22:45.957784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:22:45.959685 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:45.960135 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:45.960141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:45.960146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:45.960272 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:22:45.960574 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:22:45.960577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:22:45.960579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:22:45.960938 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:22:45.961071 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 08:22:45.961423 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 08:22:45.961718 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":100,"threads_started":1}
I20260808 08:22:45.962289 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:45.947886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36769 (local address 127.12.13.1:57317)
0808 08:22:45.948543 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:45.948559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:22:45.948664 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:22:45.950139 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:22:45.950145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:22:45.950707 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:22:45.951205 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:45.951248 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.952686 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.952689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:45.953635 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:22:45.953646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:22:45.954602 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:22:45.954606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:22:45.954807 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:22:45.955465 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:22:45.955481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:22:45.957511 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:22:45.959811 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:45.959816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:45.959818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:45.960019 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:45.960381 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:22:45.960384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:22:45.960386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:22:45.960473 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:22:45.961056 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:22:45.961059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:22:45.961154 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 08:22:45.961274 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 08:22:45.961648 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":625,"spinlock_wait_cycles":13184,"thread_start_us":96,"threads_started":1}
W20260808 08:22:45.964735 2719 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
W20260808 08:22:45.965365 2721 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:45.965456 2721 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e; no voters: 7abf678e7bdb4c14a8e86982cf95a238, c3db1b5b143b4d5f831a65c805f446ee
I20260808 08:22:45.965708 3277 raft_consensus.cc:2726] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:22:46.489200 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:46.488777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:58397)
0808 08:22:46.489040 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:46.489099 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260808 08:22:46.616526 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:46.616094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:38223)
0808 08:22:46.616368 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:46.616427 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260808 08:22:46.814688 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:46.814255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:45219)
0808 08:22:46.814536 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:46.814595 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260808 08:22:47.490624 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:47.490130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:53365)
0808 08:22:47.490450 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:47.490517 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20260808 08:22:47.617784 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:47.617277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:57407)
0808 08:22:47.617607 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:47.617675 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
I20260808 08:22:47.815953 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:47.815509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:42333)
0808 08:22:47.815790 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:47.815851 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260808 08:22:48.104020 3291 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:48.104156 3291 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:48.104663 3291 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
W20260808 08:22:48.105546 2721 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
W20260808 08:22:48.105650 2719 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:48.105830 2719 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e; no voters: 7abf678e7bdb4c14a8e86982cf95a238, c3db1b5b143b4d5f831a65c805f446ee
I20260808 08:22:48.106027 3291 raft_consensus.cc:2726] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:22:48.492004 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:48.491543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:43839)
0808 08:22:48.491828 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:48.491895 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260808 08:22:48.619199 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:48.618714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:54897)
0808 08:22:48.619018 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:48.619096 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
I20260808 08:22:48.817098 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:48.816682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:50993)
0808 08:22:48.816947 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:48.817004 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260808 08:22:49.493914 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:49.493366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:56911)
0808 08:22:49.493689 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:49.493755 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260808 08:22:49.620632 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:49.620142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:52791)
0808 08:22:49.620462 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:49.620527 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260808 08:22:49.818406 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:49.817956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:50953)
0808 08:22:49.818255 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:49.818310 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":119,"threads_started":1}
I20260808 08:22:50.495289 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:50.494836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:35611)
0808 08:22:50.495113 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:50.495179 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260808 08:22:50.622033 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:50.621487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:33207)
0808 08:22:50.621782 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:50.621922 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260808 08:22:50.819660 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:50.819237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:38391)
0808 08:22:50.819502 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:50.819560 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260808 08:22:50.845320 3308 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:50.845484 3308 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:50.846071 3308 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
W20260808 08:22:50.846938 2721 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
W20260808 08:22:50.847038 2719 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:50.847169 2719 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e; no voters: 7abf678e7bdb4c14a8e86982cf95a238, c3db1b5b143b4d5f831a65c805f446ee
I20260808 08:22:50.847361 3308 raft_consensus.cc:2726] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:22:51.496604 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:51.496141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:36877)
0808 08:22:51.496435 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:51.496500 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260808 08:22:51.623474 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:51.622894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:51139)
0808 08:22:51.623274 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:51.623338 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":155,"threads_started":1}
I20260808 08:22:51.820961 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:51.820465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:36581)
0808 08:22:51.820790 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:51.820859 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":80,"threads_started":1}
I20260808 08:22:52.497962 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:52.497403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:55355)
0808 08:22:52.497698 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:52.497761 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
I20260808 08:22:52.625041 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:52.624467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:33281)
0808 08:22:52.624812 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:52.624906 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
I20260808 08:22:52.822361 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:52.821845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:47525)
0808 08:22:52.822194 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:52.822258 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":85,"threads_started":1}
I20260808 08:22:53.499300 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:53.498867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:60249)
0808 08:22:53.499133 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:53.499197 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260808 08:22:53.626488 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:53.626042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:43899)
0808 08:22:53.626310 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:53.626380 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260808 08:22:53.823576 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:53.823154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:35859)
0808 08:22:53.823418 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:53.823474 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260808 08:22:54.448410 3321 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:54.448582 3321 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:54.449118 3321 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
W20260808 08:22:54.449994 2719 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
W20260808 08:22:54.450103 2721 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:54.450204 2721 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e; no voters: 7abf678e7bdb4c14a8e86982cf95a238, c3db1b5b143b4d5f831a65c805f446ee
I20260808 08:22:54.450385 3321 raft_consensus.cc:2726] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:22:54.500916 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:54.500376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:52223)
0808 08:22:54.500734 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:54.500805 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260808 08:22:54.628012 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:54.627493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:35963)
0808 08:22:54.627828 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:54.627892 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260808 08:22:54.824941 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:54.824458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:40191)
0808 08:22:54.824756 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:54.824825 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
I20260808 08:22:55.502367 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:55.501837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:38193)
0808 08:22:55.502171 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:55.502232 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":94,"threads_started":1}
I20260808 08:22:55.629230 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:55.628777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:39697)
0808 08:22:55.629068 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:55.629127 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
I20260808 08:22:55.826315 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:55.825761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:59681)
0808 08:22:55.826117 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:55.826198 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
I20260808 08:22:56.503724 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:56.503300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:58691)
0808 08:22:56.503562 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:56.503621 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260808 08:22:56.630820 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:56.630329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:37409)
0808 08:22:56.630653 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:56.630714 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260808 08:22:56.827684 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:56.827220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:59851)
0808 08:22:56.827523 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:56.827584 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260808 08:22:57.505045 3338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:57.504581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:45157)
0808 08:22:57.504878 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:57.504939 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260808 08:22:57.632401 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:57.631785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:52827)
0808 08:22:57.632202 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:57.632282 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260808 08:22:57.829010 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:57.828529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:56595)
0808 08:22:57.828834 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:57.828902 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
I20260808 08:22:58.443077 3341 raft_consensus.cc:482] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:22:58.443257 3341 raft_consensus.cc:504] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3db1b5b143b4d5f831a65c805f446ee" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36769 } } peers { permanent_uuid: "7abf678e7bdb4c14a8e86982cf95a238" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 39597 } } peers { permanent_uuid: "45d98465adfd4ba8a4449c6bf01f226e" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 44997 } }
I20260808 08:22:58.443756 3341 leader_election.cc:288] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769), 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597)
W20260808 08:22:58.444605 2721 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3db1b5b143b4d5f831a65c805f446ee (127.12.13.3:36769): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
W20260808 08:22:58.444715 2719 leader_election.cc:341] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7abf678e7bdb4c14a8e86982cf95a238 (127.12.13.2:39597): Not found: Tablet not found: 4b6c6f84c0d5414e9111f1eeedccac8c
I20260808 08:22:58.444867 2719 leader_election.cc:302] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d98465adfd4ba8a4449c6bf01f226e; no voters: 7abf678e7bdb4c14a8e86982cf95a238, c3db1b5b143b4d5f831a65c805f446ee
I20260808 08:22:58.445036 3341 raft_consensus.cc:2726] T 4b6c6f84c0d5414e9111f1eeedccac8c P 45d98465adfd4ba8a4449c6bf01f226e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:22:58.506455 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:58.506003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:43255)
0808 08:22:58.506287 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:58.506350 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260808 08:22:58.634068 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:58.633417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:59395)
0808 08:22:58.633796 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:58.633944 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260808 08:22:58.830483 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:58.830019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:43557)
0808 08:22:58.830316 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:58.830383 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
I20260808 08:22:59.508066 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:59.507608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.1:56437)
0808 08:22:59.507904 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:59.507963 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
I20260808 08:22:59.635552 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:59.635101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.2:46135)
0808 08:22:59.635388 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:59.635450 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260808 08:22:59.815092 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 2639
I20260808 08:22:59.829638 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 2880
I20260808 08:22:59.837167 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:22:59.833673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:43219 (local address 127.12.13.3:39905)
0808 08:22:59.836993 (+ 3320us) negotiation.cc:107] Waiting for socket to connect
0808 08:22:59.837060 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3223,"thread_start_us":1259,"threads_started":1}
I20260808 08:22:59.844308 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 3080
2026-08-08T08:22:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19865 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:22:59 dist-test-slave-6ntk krb5kdc[3351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:22:59 dist-test-slave-6ntk krb5kdc[3351](info): set up 2 sockets
Aug 08 08:22:59 dist-test-slave-6ntk krb5kdc[3351](info): commencing operation
krb5kdc: starting...
W20260808 08:23:01.162361 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:23:01 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:23:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:23:01.297160 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46083
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:23:01.415859 3376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:01.416179 3376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:01.416260 3376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:01.420238 3376 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 08:23:01.420331 3376 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 08:23:01.420359 3376 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 08:23:01.420385 3376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:23:01.420424 3376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:01.420470 3376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:01.420507 3376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:23:01.420542 3376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:23:01.425958 3376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:01.427104 3376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:01.428200 3376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:23:01.437848 3376 server_base.cc:1034] running on GCE node
W20260808 08:23:01.437708 3384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:01.437703 3387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:01.437752 3385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:01.438661 3376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:01.439966 3376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:01.441171 3376 hybrid_clock.cc:648] HybridClock initialized: now 1786177381441153 us; error 38 us; skew 500 ppm
Aug 08 08:23:01 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:01.448660 3376 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:23:01.449783 3376 webserver.cc:460] Webserver started at http://127.12.13.62:45379/ using document root <none> and password file <none>
I20260808 08:23:01.450456 3376 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:01.450526 3376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:01.450709 3376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:23:01.452911 3376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "847f456a548c492ba978c4735fae56c7"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "a474a02080453a8bbae5f25462581353"
server_key_iv: "86ece7d03aa36ac009b815c730a3fdd3"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.453478 3376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "847f456a548c492ba978c4735fae56c7"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "a474a02080453a8bbae5f25462581353"
server_key_iv: "86ece7d03aa36ac009b815c730a3fdd3"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.457743 3376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 08:23:01.460561 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:01.462126 3376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:01.462320 3376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "847f456a548c492ba978c4735fae56c7"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "a474a02080453a8bbae5f25462581353"
server_key_iv: "86ece7d03aa36ac009b815c730a3fdd3"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.462451 3376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:01.478363 3376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:01.482463 3376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:01.482741 3376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:01.492265 3376 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:46083
I20260808 08:23:01.492446 3468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:46083 every 8 connection(s)
I20260808 08:23:01.493501 3376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:23:01.494807 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 3376
I20260808 08:23:01.494933 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:23:01.495272 12340 external_mini_cluster.cc:1428] Setting key 8e5e8a0aaa6f10a190cfd87e48723979
I20260808 08:23:01.497254 3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:01.502908 3469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap starting.
Aug 08 08:23:01 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:23:01.505239 3469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:01.506106 3469 log.cc:824] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:01.508134 3469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: No bootstrap required, opened a new log
I20260808 08:23:01.510457 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:01.496805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.12.13.62:46083)
0808 08:23:01.497273 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:01.497288 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:01.497360 (+ 72us) server_negotiation.cc:407] Connection header received
0808 08:23:01.498131 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:01.498151 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:01.498496 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:01.498851 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:01.499665 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:01.500682 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:01.501558 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:01.502139 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:01.502312 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:01.504915 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:01.504920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:01.504934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:01.504963 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:01.507170 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:01.507668 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:01.507672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:01.507677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:01.507794 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:01.508107 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:01.508110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:01.508112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:01.508615 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:01.508850 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:01.509161 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 08:23:01.509616 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":151,"threads_started":1}
I20260808 08:23:01.511926 3469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:01.512176 3469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:01.512269 3469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847f456a548c492ba978c4735fae56c7, State: Initialized, Role: FOLLOWER
I20260808 08:23:01.512701 3469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:01.512859 3469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:23:01.512949 3469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:23:01.513074 3469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:01.514086 3469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:01.514438 3469 leader_election.cc:302] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 847f456a548c492ba978c4735fae56c7; no voters:
I20260808 08:23:01.514775 3469 leader_election.cc:288] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:23:01.514875 3476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:23:01.515172 3476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 LEADER]: Becoming Leader. State: Replica: 847f456a548c492ba978c4735fae56c7, State: Running, Role: LEADER
I20260808 08:23:01.515527 3476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:01.516084 3469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:23:01.516739 3476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 847f456a548c492ba978c4735fae56c7. Latest consensus state: current_term: 1 leader_uuid: "847f456a548c492ba978c4735fae56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } } }
I20260808 08:23:01.516865 3476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: This master's current role is: LEADER
I20260808 08:23:01.517335 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "847f456a548c492ba978c4735fae56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } } }
I20260808 08:23:01.517459 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: This master's current role is: LEADER
I20260808 08:23:01.518505 3488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:23:01.521629 3488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:23:01.527302 3488 catalog_manager.cc:1347] Generated new cluster ID: 50199b3ef9c9411c8f310f35370b50f1
I20260808 08:23:01.527385 3488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:23:01.547227 3488 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:23:01.548295 3488 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:23:01.555644 3488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Generated new TSK 0
I20260808 08:23:01.556478 3488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:23:01.595142 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46083
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:23:01.717793 3509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:01.718102 3509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:01.718191 3509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:01.722259 3509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:01.722333 3509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:01.722438 3509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:23:01.727816 3509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:01.729007 3509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:01.730149 3509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:01.740649 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:01.740737 3509 server_base.cc:1034] running on GCE node
W20260808 08:23:01.740643 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:01.740655 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:01.741287 3509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:01.742053 3509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:01.743221 3509 hybrid_clock.cc:648] HybridClock initialized: now 1786177381743170 us; error 66 us; skew 500 ppm
Aug 08 08:23:01 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:01.746280 3509 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:23:01.747262 3509 webserver.cc:460] Webserver started at http://127.12.13.1:38917/ using document root <none> and password file <none>
I20260808 08:23:01.747895 3509 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:01.747989 3509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:01.748236 3509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:23:01.750471 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "1f8bf840ed95407cae38f025aa8c0c4a"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "d1676f569bfcb8edb95fe70bca61c269"
server_key_iv: "06ec1683a8b933abcad7f6e1bbd95c86"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.751065 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "1f8bf840ed95407cae38f025aa8c0c4a"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "d1676f569bfcb8edb95fe70bca61c269"
server_key_iv: "06ec1683a8b933abcad7f6e1bbd95c86"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.755288 3509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:23:01.758831 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:01.760191 3509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:23:01.760373 3509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "1f8bf840ed95407cae38f025aa8c0c4a"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "d1676f569bfcb8edb95fe70bca61c269"
server_key_iv: "06ec1683a8b933abcad7f6e1bbd95c86"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.760514 3509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:01.777848 3509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:01.780898 3509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:01.781109 3509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:01.781764 3509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:01.782673 3509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:01.782745 3509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:01.782819 3509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:01.782874 3509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:01.792220 3509 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:46109
I20260808 08:23:01.792243 3700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:46109 every 8 connection(s)
I20260808 08:23:01.793126 3509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:23:01.793495 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 3509
I20260808 08:23:01.793632 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:23:01.794036 12340 external_mini_cluster.cc:1428] Setting key fb4d457cb1d692c79375cd21e04be843
Aug 08 08:23:01 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:23:01.808522 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:01.795067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:55371 (local address 127.12.13.62:46083)
0808 08:23:01.795209 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:01.795214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:01.796147 (+ 933us) server_negotiation.cc:407] Connection header received
0808 08:23:01.797485 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:01.797490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:01.797555 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:01.797654 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:01.799355 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:01.800247 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:01.801382 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:01.801853 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:01.802003 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:01.804712 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:01.804718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:01.804723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:01.804756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:01.806498 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:01.807021 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:01.807023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:01.807025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:01.807101 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:01.807532 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:01.807535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:01.807536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:01.807714 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:01.807795 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:01.808283 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 08:23:01.808394 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:23:01.809597 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:01.795465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:55371)
0808 08:23:01.795965 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:01.796019 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:01.797263 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:01.797907 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:01.797917 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:01.798456 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:01.799181 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:01.799203 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:01.800380 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:01.800383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:01.801255 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:01.801264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:01.802036 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:01.802039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:01.802181 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:01.802832 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:01.802855 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:01.804534 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:01.806606 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:01.806611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:01.806623 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:01.806930 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:01.807197 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:01.807199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:01.807201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:01.807435 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:01.807807 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:01.807812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:01.808065 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 08:23:01.808220 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 08:23:01.808524 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20260808 08:23:01.810945 3701 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46083
I20260808 08:23:01.811182 3701 heartbeater.cc:464] Registering TS with master...
I20260808 08:23:01.811815 3701 heartbeater.cc:511] Master 127.12.13.62:46083 requested a full tablet report, sending...
I20260808 08:23:01.813695 3422 ts_manager.cc:194] Registered new tserver with Master: 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109)
I20260808 08:23:01.815491 3422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:55371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:23:01.829954 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46083
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:23:01.949930 3707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:01.950237 3707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:01.950284 3707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:01.954381 3707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:01.954448 3707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:01.954509 3707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:23:01.959923 3707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:01.961163 3707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:01.962296 3707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:01.972916 3716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:01.972947 3712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:01.973229 3714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:01.973033 3707 server_base.cc:1034] running on GCE node
I20260808 08:23:01.973662 3707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:01.974421 3707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:01.975644 3707 hybrid_clock.cc:648] HybridClock initialized: now 1786177381975606 us; error 48 us; skew 500 ppm
Aug 08 08:23:01 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:01.979087 3707 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:23:01.980070 3707 webserver.cc:460] Webserver started at http://127.12.13.2:35575/ using document root <none> and password file <none>
I20260808 08:23:01.980715 3707 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:01.980813 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:01.981066 3707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:23:01.983299 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "7f19dc4aa4594e019378967a7cbb4df3"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "1959be2ec989b4b58dec5f72f9912313"
server_key_iv: "914308c90dfe641fb048e85ca5b0c3c9"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.983901 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "7f19dc4aa4594e019378967a7cbb4df3"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "1959be2ec989b4b58dec5f72f9912313"
server_key_iv: "914308c90dfe641fb048e85ca5b0c3c9"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.988085 3707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:23:01.990760 3726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:01.992105 3707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:01.992254 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "7f19dc4aa4594e019378967a7cbb4df3"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "1959be2ec989b4b58dec5f72f9912313"
server_key_iv: "914308c90dfe641fb048e85ca5b0c3c9"
server_key_version: "encryptionkey@0"
I20260808 08:23:01.992384 3707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:02.006150 3707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:02.009987 3707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:02.010218 3707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:02.010931 3707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:02.011871 3707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:02.011957 3707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:02.012060 3707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:02.012112 3707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:02.022089 3707 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:37073
I20260808 08:23:02.022110 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:37073 every 8 connection(s)
I20260808 08:23:02.023084 3707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:23:02.028242 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 3707
I20260808 08:23:02.028352 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:23:02.028656 12340 external_mini_cluster.cc:1428] Setting key 33739404e3a39e9fa7c67558d3bb0939
Aug 08 08:23:02 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:23:02.039372 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.025030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:53557 (local address 127.12.13.62:46083)
0808 08:23:02.025184 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.025189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.026245 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 08:23:02.027684 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.027687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.027740 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.027821 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:02.029223 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.029953 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.031698 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.031926 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.032014 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.035324 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:02.035338 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:02.035340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:02.035367 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:02.037034 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:02.037601 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:02.037605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:02.037607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:02.037694 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:02.038131 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:02.038136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:02.038137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:02.038352 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:02.038470 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.039040 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.039179 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:23:02.040316 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.025507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:53557)
0808 08:23:02.026079 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.026128 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.027467 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.027984 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.027991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.028447 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:02.029047 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.029068 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.030112 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.030115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.031559 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.031574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.032097 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.032104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.032348 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.033099 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:02.033116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:02.035143 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:02.037168 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:02.037174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:02.037192 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:02.037489 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:02.037793 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:02.037797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:02.037798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:02.038027 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:02.038471 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:02.038479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:02.038764 (+ 285us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.038974 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.039342 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260808 08:23:02.041596 3900 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46083
I20260808 08:23:02.041914 3900 heartbeater.cc:464] Registering TS with master...
I20260808 08:23:02.042496 3900 heartbeater.cc:511] Master 127.12.13.62:46083 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:23:02.043697 3422 ts_manager.cc:194] Registered new tserver with Master: 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
I20260808 08:23:02.044382 3422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:53557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:23:02.062256 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46083
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:23:02.190425 3907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:02.190835 3907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:02.190960 3907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:02.196718 3907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:02.196790 3907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:02.196878 3907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:23:02.202481 3907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:02.203636 3907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:02.204717 3907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:02.215175 3915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:02.215188 3920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:02.215312 3916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:02.215502 3907 server_base.cc:1034] running on GCE node
I20260808 08:23:02.216070 3907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:02.216915 3907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:02.218137 3907 hybrid_clock.cc:648] HybridClock initialized: now 1786177382218131 us; error 61 us; skew 500 ppm
Aug 08 08:23:02 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:02.221566 3907 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:23:02.222644 3907 webserver.cc:460] Webserver started at http://127.12.13.3:34809/ using document root <none> and password file <none>
I20260808 08:23:02.223274 3907 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:02.223371 3907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:02.223605 3907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:23:02.225716 3907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "922b6a3ea0674f8f905b47f4f689703d"
format_stamp: "Formatted at 2026-08-08 08:23:02 on dist-test-slave-6ntk"
server_key: "99f248287c7bc6fc16f3c07407e52cac"
server_key_iv: "55613d57f5bd860ca115a1abd479a89b"
server_key_version: "encryptionkey@0"
I20260808 08:23:02.226426 3907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "922b6a3ea0674f8f905b47f4f689703d"
format_stamp: "Formatted at 2026-08-08 08:23:02 on dist-test-slave-6ntk"
server_key: "99f248287c7bc6fc16f3c07407e52cac"
server_key_iv: "55613d57f5bd860ca115a1abd479a89b"
server_key_version: "encryptionkey@0"
I20260808 08:23:02.230448 3907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:23:02.233223 3933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:02.234742 3907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:02.234908 3907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "922b6a3ea0674f8f905b47f4f689703d"
format_stamp: "Formatted at 2026-08-08 08:23:02 on dist-test-slave-6ntk"
server_key: "99f248287c7bc6fc16f3c07407e52cac"
server_key_iv: "55613d57f5bd860ca115a1abd479a89b"
server_key_version: "encryptionkey@0"
I20260808 08:23:02.235045 3907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:02.248207 3907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:02.251772 3907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:02.252017 3907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:02.252686 3907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:02.253613 3907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:02.253686 3907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:02.253762 3907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:02.253813 3907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:02.264286 3907 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:33199
I20260808 08:23:02.264390 4112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:33199 every 8 connection(s)
I20260808 08:23:02.265368 3907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:23:02.270992 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 3907
I20260808 08:23:02.271140 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:23:02.271445 12340 external_mini_cluster.cc:1428] Setting key b3d862025651ecd63cd9ea5e2dcf0686
Aug 08 08:23:02 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:23:02.280269 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.267517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:46825 (local address 127.12.13.62:46083)
0808 08:23:02.267680 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.267685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.268737 (+ 1052us) server_negotiation.cc:407] Connection header received
0808 08:23:02.270067 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.270069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.270119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.270197 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:02.271548 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.272166 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.273856 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.274134 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.274222 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.276947 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:02.276949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:02.276952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:02.276977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:02.278376 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:02.278877 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:02.278879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:02.278881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:02.278959 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:02.279276 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:02.279277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:02.279279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:02.279468 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:02.279549 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.280038 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.280142 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 08:23:02.281231 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.268039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:46825)
0808 08:23:02.268572 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.268618 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.269845 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.270401 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.270407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.270793 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:02.271383 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.271401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.272300 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.272305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.273715 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.273729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.274276 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.274279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.274460 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.275203 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:02.275220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:02.276775 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:02.278489 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:02.278494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:02.278508 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:02.278772 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:02.279043 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:02.279046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:02.279047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:02.279186 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:02.279562 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:02.279571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:02.279823 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.279985 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.280249 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":133,"threads_started":1}
I20260808 08:23:02.282579 4114 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46083
I20260808 08:23:02.282840 4114 heartbeater.cc:464] Registering TS with master...
I20260808 08:23:02.283402 4114 heartbeater.cc:511] Master 127.12.13.62:46083 requested a full tablet report, sending...
I20260808 08:23:02.284675 3422 ts_manager.cc:194] Registered new tserver with Master: 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199)
I20260808 08:23:02.285384 3422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:46825
I20260808 08:23:02.285759 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:23:02.295636 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.287901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.12.13.62:46083)
0808 08:23:02.288033 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.288037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.288251 (+ 214us) server_negotiation.cc:407] Connection header received
0808 08:23:02.288335 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.288338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.288378 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.288443 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:02.289090 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.289575 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.290785 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.290997 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.291077 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.292681 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:02.292684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:02.292686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:02.292708 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:02.294050 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:02.294499 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:02.294502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:02.294504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:02.294594 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:02.294840 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:02.294843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:02.294844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:02.295066 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:02.295172 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.295374 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.295466 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 08:23:02.298537 3422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:23:02.301645 3422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:23:02.316712 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.311487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.12.13.3:33199)
0808 08:23:02.312634 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.312644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.312661 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:23:02.312720 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.312723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.312855 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.313002 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:02.313219 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.314014 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.315291 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.315966 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.316090 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.316252 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.316445 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.316544 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":122,"threads_started":1}
I20260808 08:23:02.316725 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.311315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:33199 (local address 127.0.0.1:36050)
0808 08:23:02.311688 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.311702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.311780 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.312993 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.312997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.313015 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:02.313123 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.313133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.314141 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.314145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.315164 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.315176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.316090 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.316093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.316226 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.316269 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.316412 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.316537 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260808 08:23:02.317338 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.310592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46109 (local address 127.0.0.1:33596)
0808 08:23:02.310951 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.311008 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.311144 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.311757 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.311764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.311782 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:02.311981 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.311989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.313572 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.313576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.314960 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.314969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.316125 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.316132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.316985 (+ 853us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.317035 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.317108 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.317182 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":728,"thread_start_us":75,"threads_started":1}
I20260808 08:23:02.317705 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.311659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:37073 (local address 127.0.0.1:59666)
0808 08:23:02.312199 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.312211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.312399 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.313098 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.313101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.313121 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:02.313209 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.313215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.314881 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.314885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.316086 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.316099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.317263 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.317268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.317356 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.317379 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.317461 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.317547 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260808 08:23:02.317945 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.310779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.12.13.1:46109)
0808 08:23:02.311089 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.311097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.311122 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:23:02.311336 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.311342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.311575 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.312149 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:02.312314 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.313438 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.315105 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.315986 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.316226 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.316402 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.317260 (+ 858us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.317381 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260808 08:23:02.318552 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.312075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.12.13.2:37073)
0808 08:23:02.312624 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.312636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.312659 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:23:02.312736 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.312741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.312938 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.313335 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:02.313496 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.314717 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.316232 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.317118 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.318056 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.318215 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.318266 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.318355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":154,"threads_started":1}
I20260808 08:23:02.320307 4008 tablet_service.cc:1467] Processing CreateTablet for tablet cbae5a2b25404020b1f1599f932660f6 (DEFAULT_TABLE table=test-table [id=58785774a6a84aebb828c77d7928afae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:23:02.320806 3798 tablet_service.cc:1467] Processing CreateTablet for tablet cbae5a2b25404020b1f1599f932660f6 (DEFAULT_TABLE table=test-table [id=58785774a6a84aebb828c77d7928afae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:23:02.320819 3601 tablet_service.cc:1467] Processing CreateTablet for tablet cbae5a2b25404020b1f1599f932660f6 (DEFAULT_TABLE table=test-table [id=58785774a6a84aebb828c77d7928afae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:23:02.321583 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbae5a2b25404020b1f1599f932660f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:02.321856 3601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbae5a2b25404020b1f1599f932660f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:02.321915 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbae5a2b25404020b1f1599f932660f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:02.328285 4140 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Bootstrap starting.
I20260808 08:23:02.329528 4142 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Bootstrap starting.
I20260808 08:23:02.330169 4141 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap starting.
I20260808 08:23:02.331370 4142 tablet_bootstrap.cc:654] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:02.331355 4140 tablet_bootstrap.cc:654] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:02.332365 4141 tablet_bootstrap.cc:654] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:02.332361 4140 log.cc:824] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:02.332557 4142 log.cc:824] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:02.333290 4141 log.cc:824] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:02.334626 4140 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: No bootstrap required, opened a new log
I20260808 08:23:02.334875 4142 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: No bootstrap required, opened a new log
I20260808 08:23:02.334878 4140 ts_tablet_manager.cc:1397] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 08:23:02.335083 4142 ts_tablet_manager.cc:1397] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:23:02.335474 4141 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: No bootstrap required, opened a new log
I20260808 08:23:02.335654 4141 ts_tablet_manager.cc:1397] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 08:23:02.338219 4140 raft_consensus.cc:348] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.338581 4140 raft_consensus.cc:374] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:02.338663 4140 raft_consensus.cc:729] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f19dc4aa4594e019378967a7cbb4df3, State: Initialized, Role: FOLLOWER
I20260808 08:23:02.339282 4140 consensus_queue.cc:260] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.339290 4142 raft_consensus.cc:348] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.339609 4142 raft_consensus.cc:374] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:02.339704 4142 raft_consensus.cc:729] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922b6a3ea0674f8f905b47f4f689703d, State: Initialized, Role: FOLLOWER
I20260808 08:23:02.340215 4142 consensus_queue.cc:260] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.340503 4140 ts_tablet_manager.cc:1428] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 08:23:02.340520 4141 raft_consensus.cc:348] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.340946 4114 heartbeater.cc:503] Master 127.12.13.62:46083 was elected leader, sending a full tablet report...
I20260808 08:23:02.340960 3900 heartbeater.cc:503] Master 127.12.13.62:46083 was elected leader, sending a full tablet report...
I20260808 08:23:02.341244 4142 ts_tablet_manager.cc:1428] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:23:02.341387 4141 raft_consensus.cc:374] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:02.341456 4141 raft_consensus.cc:729] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8bf840ed95407cae38f025aa8c0c4a, State: Initialized, Role: FOLLOWER
I20260808 08:23:02.341914 4141 consensus_queue.cc:260] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.342900 4141 ts_tablet_manager.cc:1428] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 08:23:02.343273 3701 heartbeater.cc:503] Master 127.12.13.62:46083 was elected leader, sending a full tablet report...
I20260808 08:23:02.343364 4149 raft_consensus.cc:482] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:02.343504 4149 raft_consensus.cc:504] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.345640 4149 leader_election.cc:288] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109), 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199)
I20260808 08:23:02.349602 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.345053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46109 (local address 127.12.13.2:48535)
0808 08:23:02.345331 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.345347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.345436 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.345932 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.345935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.345958 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:02.346091 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.346104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.347084 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.347088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.348323 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.348335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.349152 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.349156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.349262 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.349284 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.349393 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.349466 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":3}
I20260808 08:23:02.349597 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.345302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:48535 (local address 127.12.13.1:46109)
0808 08:23:02.345520 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.345525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.345540 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:23:02.345584 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.345587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.345638 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.345711 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:02.346208 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.346967 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.348468 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.349023 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.349139 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.349204 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.349408 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.349479 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260808 08:23:02.350291 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.346338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:52911 (local address 127.12.13.3:33199)
0808 08:23:02.346462 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.346467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.346674 (+ 207us) server_negotiation.cc:407] Connection header received
0808 08:23:02.346842 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.346845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.346885 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.346927 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:02.347362 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.347959 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.349210 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.349627 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.349752 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.349822 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.350112 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.350186 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 08:23:02.350283 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.345926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:33199 (local address 127.12.13.2:52911)
0808 08:23:02.346596 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.346609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:02.346741 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:02.347057 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:02.347061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:02.347079 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:02.347225 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.347230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.348099 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.348105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.349083 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:02.349094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.349833 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.349838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:02.349973 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.349996 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:23:02.350078 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 08:23:02.350145 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":24,"thread_start_us":128,"threads_started":1}
I20260808 08:23:02.350432 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbae5a2b25404020b1f1599f932660f6" candidate_uuid: "7f19dc4aa4594e019378967a7cbb4df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" is_pre_election: true
I20260808 08:23:02.350755 3635 raft_consensus.cc:2454] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f19dc4aa4594e019378967a7cbb4df3 in term 0.
I20260808 08:23:02.350901 4035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbae5a2b25404020b1f1599f932660f6" candidate_uuid: "7f19dc4aa4594e019378967a7cbb4df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922b6a3ea0674f8f905b47f4f689703d" is_pre_election: true
I20260808 08:23:02.351202 4035 raft_consensus.cc:2454] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f19dc4aa4594e019378967a7cbb4df3 in term 0.
I20260808 08:23:02.351296 3730 leader_election.cc:302] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a, 7f19dc4aa4594e019378967a7cbb4df3; no voters:
I20260808 08:23:02.351601 4149 raft_consensus.cc:2781] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:23:02.351703 4149 raft_consensus.cc:482] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:23:02.351751 4149 raft_consensus.cc:3037] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:02.352849 4149 raft_consensus.cc:504] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.353193 4149 leader_election.cc:288] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [CANDIDATE]: Term 1 election: Requested vote from peers 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109), 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199)
I20260808 08:23:02.353585 4035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbae5a2b25404020b1f1599f932660f6" candidate_uuid: "7f19dc4aa4594e019378967a7cbb4df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922b6a3ea0674f8f905b47f4f689703d"
I20260808 08:23:02.353699 4035 raft_consensus.cc:3037] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:02.353598 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbae5a2b25404020b1f1599f932660f6" candidate_uuid: "7f19dc4aa4594e019378967a7cbb4df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8bf840ed95407cae38f025aa8c0c4a"
I20260808 08:23:02.353828 3635 raft_consensus.cc:3037] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:02.354820 4035 raft_consensus.cc:2454] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f19dc4aa4594e019378967a7cbb4df3 in term 1.
I20260808 08:23:02.355147 3730 leader_election.cc:302] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f19dc4aa4594e019378967a7cbb4df3, 922b6a3ea0674f8f905b47f4f689703d; no voters:
I20260808 08:23:02.355248 3635 raft_consensus.cc:2454] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f19dc4aa4594e019378967a7cbb4df3 in term 1.
I20260808 08:23:02.355314 4149 raft_consensus.cc:2781] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:23:02.355438 4149 raft_consensus.cc:686] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f19dc4aa4594e019378967a7cbb4df3, State: Running, Role: LEADER
I20260808 08:23:02.355971 4149 consensus_queue.cc:237] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:02.359472 3420 catalog_manager.cc:5495] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2). New cstate: current_term: 1 leader_uuid: "7f19dc4aa4594e019378967a7cbb4df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } health_report { overall_health: HEALTHY } } }
I20260808 08:23:02.380288 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.376913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.12.13.2:37073)
0808 08:23:02.377055 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:02.377059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:02.377073 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:23:02.377170 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:02.377173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:02.377228 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:02.377314 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:23:02.377557 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.378091 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.378933 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:02.379135 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:02.379208 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:02.379491 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:23:02.379914 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:23:02.380003 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:02.380060 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 08:23:02.380106 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:23:02.386546 4035 raft_consensus.cc:1261] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Refusing update from remote peer 7f19dc4aa4594e019378967a7cbb4df3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:23:02.386582 3635 raft_consensus.cc:1261] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Refusing update from remote peer 7f19dc4aa4594e019378967a7cbb4df3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:23:02.387298 4149 consensus_queue.cc:1035] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:23:02.387570 4152 consensus_queue.cc:1035] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:23:02.405227 3422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:23:02.405426 3422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:23:02.407963 3422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Sending DeleteTablet for 3 replicas of tablet cbae5a2b25404020b1f1599f932660f6
I20260808 08:23:02.408825 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 3509
I20260808 08:23:02.408787 3601 tablet_service.cc:1514] Processing DeleteTablet for tablet cbae5a2b25404020b1f1599f932660f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:33596
I20260808 08:23:02.408857 3798 tablet_service.cc:1514] Processing DeleteTablet for tablet cbae5a2b25404020b1f1599f932660f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:59666
I20260808 08:23:02.415330 4008 tablet_service.cc:1514] Processing DeleteTablet for tablet cbae5a2b25404020b1f1599f932660f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:36050
I20260808 08:23:02.416059 4170 tablet_replica.cc:330] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: stopping tablet replica
I20260808 08:23:02.416425 4170 raft_consensus.cc:2229] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:23:02.416746 4170 raft_consensus.cc:2258] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:23:02.420647 4168 tablet_replica.cc:330] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: stopping tablet replica
I20260808 08:23:02.420991 4168 raft_consensus.cc:2229] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:23:02.421366 4170 ts_tablet_manager.cc:1905] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:23:02.421433 4168 raft_consensus.cc:2258] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:23:02.425542 4170 ts_tablet_manager.cc:1918] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:23:02.425622 4170 log.cc:1196] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/cbae5a2b25404020b1f1599f932660f6
I20260808 08:23:02.425963 4170 ts_tablet_manager.cc:1939] T cbae5a2b25404020b1f1599f932660f6 P 922b6a3ea0674f8f905b47f4f689703d: Deleting consensus metadata
I20260808 08:23:02.427445 3404 catalog_manager.cc:4841] TS 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): tablet cbae5a2b25404020b1f1599f932660f6 (table test-table [id=58785774a6a84aebb828c77d7928afae]) successfully deleted
I20260808 08:23:02.428216 4168 ts_tablet_manager.cc:1905] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:23:02.430476 4168 ts_tablet_manager.cc:1918] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:23:02.430595 4168 log.cc:1196] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/cbae5a2b25404020b1f1599f932660f6
I20260808 08:23:02.430948 4168 ts_tablet_manager.cc:1939] T cbae5a2b25404020b1f1599f932660f6 P 7f19dc4aa4594e019378967a7cbb4df3: Deleting consensus metadata
I20260808 08:23:02.432041 3404 catalog_manager.cc:4841] TS 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): tablet cbae5a2b25404020b1f1599f932660f6 (table test-table [id=58785774a6a84aebb828c77d7928afae]) successfully deleted
W20260808 08:23:02.432780 3404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:46109 (error 108)
I20260808 08:23:02.433038 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 3707
I20260808 08:23:02.433965 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:02.433519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46109 (local address 127.0.0.1:33606)
0808 08:23:02.433730 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:02.433829 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260808 08:23:02.434182 3404 catalog_manager.cc:4584] TS 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbae5a2b25404020b1f1599f932660f6: Network error: Client connection negotiation failed: client connection to 127.12.13.1:46109: connect: Connection refused (error 111)
I20260808 08:23:02.449676 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 3907
I20260808 08:23:02.465301 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 3376
I20260808 08:23:22.478394 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46083
--webserver_interface=127.12.13.62
--webserver_port=45379
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:23:22.598099 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:22.598382 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:22.598459 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:22.602555 4172 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 08:23:22.602612 4172 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 08:23:22.602629 4172 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 08:23:22.602643 4172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:23:22.602662 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:22.602732 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:22.602761 4172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:23:22.602797 4172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:23:22.608297 4172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:22.609478 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:22.610625 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:22.625481 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:22.625557 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:22.625560 4172 server_base.cc:1034] running on GCE node
W20260808 08:23:22.625768 4187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:22.626683 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:22.627950 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:22.629170 4172 hybrid_clock.cc:648] HybridClock initialized: now 1786177402629155 us; error 33 us; skew 500 ppm
Aug 08 08:23:22 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177402, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:22.636590 4172 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:23:22.637694 4172 webserver.cc:460] Webserver started at http://127.12.13.62:45379/ using document root <none> and password file <none>
I20260808 08:23:22.638425 4172 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:22.638549 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:22.642967 4172 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:23:22.645637 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:22.647297 4172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:23:22.647485 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "847f456a548c492ba978c4735fae56c7"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "a474a02080453a8bbae5f25462581353"
server_key_iv: "86ece7d03aa36ac009b815c730a3fdd3"
server_key_version: "encryptionkey@0"
I20260808 08:23:22.648149 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:22.669930 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:22.673108 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:22.673346 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:22.681233 4172 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:46083
I20260808 08:23:22.681540 4270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:46083 every 8 connection(s)
I20260808 08:23:22.682456 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:23:22.687077 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 4172
I20260808 08:23:22.687422 4271 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:23:22.687611 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:46109
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:46083
--webserver_port=38917
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:23:22.690097 4271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap starting.
I20260808 08:23:22.715873 4271 log.cc:824] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:22.726598 4271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:23:22.727013 4271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap complete.
I20260808 08:23:22.731374 4271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:22.731828 4271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847f456a548c492ba978c4735fae56c7, State: Initialized, Role: FOLLOWER
I20260808 08:23:22.732336 4271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:22.732483 4271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:23:22.732580 4271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:23:22.732705 4271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 1 FOLLOWER]: Advancing to term 2
I20260808 08:23:22.734812 4271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:22.735160 4271 leader_election.cc:302] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 847f456a548c492ba978c4735fae56c7; no voters:
I20260808 08:23:22.735504 4271 leader_election.cc:288] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 08:23:22.735643 4276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 08:23:22.735913 4276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 LEADER]: Becoming Leader. State: Replica: 847f456a548c492ba978c4735fae56c7, State: Running, Role: LEADER
I20260808 08:23:22.736305 4276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:22.736727 4271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:23:22.737357 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 847f456a548c492ba978c4735fae56c7. Latest consensus state: current_term: 2 leader_uuid: "847f456a548c492ba978c4735fae56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } } }
I20260808 08:23:22.737663 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: This master's current role is: LEADER
I20260808 08:23:22.737793 4277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "847f456a548c492ba978c4735fae56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } } }
I20260808 08:23:22.738013 4277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: This master's current role is: LEADER
I20260808 08:23:22.742255 4297 catalog_manager.cc:1259] Loaded cluster ID: 50199b3ef9c9411c8f310f35370b50f1
I20260808 08:23:22.742324 4297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: loading cluster ID for follower catalog manager: success
I20260808 08:23:22.744843 4297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: acquiring CA information for follower catalog manager: success
I20260808 08:23:22.746513 4297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:23:22.746650 4287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:23:22.749147 4287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:23:22.749699 4287 catalog_manager.cc:1259] Loaded cluster ID: 50199b3ef9c9411c8f310f35370b50f1
I20260808 08:23:22.749791 4287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:23:22.750543 4287 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:23:22.751062 4287 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Loaded TSK: 0
I20260808 08:23:22.765609 4287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Generated new TSK 1
I20260808 08:23:22.765982 4287 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 08:23:22.823652 4273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:22.824033 4273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:22.824143 4273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:22.828608 4273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:22.828717 4273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:22.828816 4273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:23:22.834630 4273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:46109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=38917
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:22.835848 4273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:22.837010 4273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:22.849056 4309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:22.849058 4308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:22.849059 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:22.849642 4273 server_base.cc:1034] running on GCE node
I20260808 08:23:22.850173 4273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:22.850922 4273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:22.852111 4273 hybrid_clock.cc:648] HybridClock initialized: now 1786177402852077 us; error 48 us; skew 500 ppm
Aug 08 08:23:22 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177402, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:22.855237 4273 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:23:22.856345 4273 webserver.cc:460] Webserver started at http://127.12.13.1:38917/ using document root <none> and password file <none>
I20260808 08:23:22.856956 4273 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:22.857033 4273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:22.861399 4273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:23:22.863933 4321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:22.865399 4273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:23:22.865506 4273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "1f8bf840ed95407cae38f025aa8c0c4a"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "d1676f569bfcb8edb95fe70bca61c269"
server_key_iv: "06ec1683a8b933abcad7f6e1bbd95c86"
server_key_version: "encryptionkey@0"
I20260808 08:23:22.866250 4273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:22.908254 4273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:22.912191 4273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:22.912436 4273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:22.913160 4273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:22.914513 4336 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 08:23:22.918948 4273 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 08:23:22.919014 4273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260808 08:23:22.919091 4273 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 08:23:22.920941 4273 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 08:23:22.920992 4273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 08:23:22.921360 4336 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap starting.
I20260808 08:23:22.932893 4273 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:46109
I20260808 08:23:22.932981 4489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:46109 every 8 connection(s)
I20260808 08:23:22.934053 4273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:23:22.937476 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 4273
I20260808 08:23:22.938094 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:37073
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:46083
--webserver_port=35575
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:23:22.943266 4336 log.cc:824] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:23:22 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177402, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:23:22.953281 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:22.936492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:42263)
0808 08:23:22.937200 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:22.937282 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:22.938501 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:22.940065 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:22.940074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:22.940704 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:22.941442 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:22.941465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.943056 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.943059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:22.944264 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:22.944277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.945143 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.945147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:22.945343 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:22.946039 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:22.946057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:22.947606 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:22.949858 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:22.949902 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:22.949916 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:22.950200 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:22.950596 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:22.950599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:22.950601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:22.950756 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:22.951378 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:22.951393 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:22.951735 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 08:23:22.951939 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 08:23:22.952303 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":184,"threads_started":1}
I20260808 08:23:22.953269 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:22.936504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:42263 (local address 127.12.13.62:46083)
0808 08:23:22.936950 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:22.936972 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:22.937711 (+ 739us) server_negotiation.cc:407] Connection header received
0808 08:23:22.939117 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:22.939136 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:22.939578 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:22.939989 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:22.941666 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.942846 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.944415 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.944975 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.945176 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:22.947827 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:22.947832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:22.947846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:22.947877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:22.949701 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:22.950342 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:22.950348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:22.950352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:22.950478 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:22.950876 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:22.950878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:22.950885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:22.951250 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:22.951405 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:22.952083 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 08:23:22.952362 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
I20260808 08:23:22.953979 4336 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:23:22.954628 4336 tablet_bootstrap.cc:492] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap complete.
I20260808 08:23:22.956137 4492 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46083
I20260808 08:23:22.956315 4336 ts_tablet_manager.cc:1397] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20260808 08:23:22.956416 4492 heartbeater.cc:464] Registering TS with master...
I20260808 08:23:22.956931 4492 heartbeater.cc:511] Master 127.12.13.62:46083 requested a full tablet report, sending...
I20260808 08:23:22.959403 4220 ts_manager.cc:194] Registered new tserver with Master: 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109)
I20260808 08:23:22.959371 4336 raft_consensus.cc:348] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:22.959916 4336 raft_consensus.cc:729] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8bf840ed95407cae38f025aa8c0c4a, State: Initialized, Role: FOLLOWER
I20260808 08:23:22.960448 4336 consensus_queue.cc:260] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } } peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } }
I20260808 08:23:22.961606 4336 ts_tablet_manager.cc:1428] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 08:23:22.963291 4220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:42263
I20260808 08:23:22.966333 4492 heartbeater.cc:503] Master 127.12.13.62:46083 was elected leader, sending a full tablet report...
I20260808 08:23:22.969843 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:22.962769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46109 (local address 127.0.0.1:34152)
0808 08:23:22.964245 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:22.964280 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:22.964427 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:22.965506 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:22.965512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:22.965539 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:22.965686 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:22.965693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.966930 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.966935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:22.968275 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:22.968288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.969362 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.969366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:22.969484 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:22.969517 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 08:23:22.969615 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 08:23:22.969719 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"thread_start_us":125,"threads_started":1}
I20260808 08:23:22.969936 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:22.962856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.12.13.1:46109)
0808 08:23:22.964747 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:22.964759 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:22.964784 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:23:22.964894 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:22.964899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:22.965311 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:22.965479 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:22.965819 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.966801 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.968418 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:22.969202 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:22.969321 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:22.969464 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:22.969667 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 08:23:22.969764 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":108,"threads_started":1}
I20260808 08:23:22.971145 4394 tablet_service.cc:1514] Processing DeleteTablet for tablet cbae5a2b25404020b1f1599f932660f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:34152
I20260808 08:23:22.972252 4503 tablet_replica.cc:330] stopping tablet replica
I20260808 08:23:22.972591 4503 raft_consensus.cc:2229] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:23:22.972892 4503 raft_consensus.cc:2258] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:23:22.973989 4503 ts_tablet_manager.cc:1905] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:23:22.977711 4503 ts_tablet_manager.cc:1918] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:23:22.977833 4503 log.cc:1196] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal/wals/cbae5a2b25404020b1f1599f932660f6
I20260808 08:23:22.978142 4503 ts_tablet_manager.cc:1939] T cbae5a2b25404020b1f1599f932660f6 P 1f8bf840ed95407cae38f025aa8c0c4a: Deleting consensus metadata
I20260808 08:23:22.980088 4201 catalog_manager.cc:4841] TS 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109): tablet cbae5a2b25404020b1f1599f932660f6 (table test-table [id=58785774a6a84aebb828c77d7928afae]) successfully deleted
W20260808 08:23:23.074642 4496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:23.074949 4496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:23.074995 4496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:23.079336 4496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:23.079412 4496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:23.079516 4496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:23:23.085253 4496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:37073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=35575
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:23.086575 4496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:23.087785 4496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:23:23.099048 4496 server_base.cc:1034] running on GCE node
W20260808 08:23:23.098999 4514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:23.099071 4512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:23.098999 4516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:23.099812 4496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:23.100550 4496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:23.101723 4496 hybrid_clock.cc:648] HybridClock initialized: now 1786177403101705 us; error 29 us; skew 500 ppm
Aug 08 08:23:23 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177403, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:23.104861 4496 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:23:23.105803 4496 webserver.cc:460] Webserver started at http://127.12.13.2:35575/ using document root <none> and password file <none>
I20260808 08:23:23.106521 4496 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:23.106673 4496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:23.111289 4496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:23:23.113652 4528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:23.115128 4496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:23.115273 4496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "7f19dc4aa4594e019378967a7cbb4df3"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "1959be2ec989b4b58dec5f72f9912313"
server_key_iv: "914308c90dfe641fb048e85ca5b0c3c9"
server_key_version: "encryptionkey@0"
I20260808 08:23:23.115878 4496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:23.132200 4496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:23.136060 4496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:23.136303 4496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:23.137101 4496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:23.138294 4496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:23.138347 4496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:23.138425 4496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:23.138456 4496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:23.148600 4496 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:37073
I20260808 08:23:23.148619 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:37073 every 8 connection(s)
I20260808 08:23:23.149713 4496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:23:23.158213 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 4496
I20260808 08:23:23.158722 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:33199
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:46083
--webserver_port=34809
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 08:23:23 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177403, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:23:23.165787 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.151761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:37459 (local address 127.12.13.62:46083)
0808 08:23:23.152000 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.152004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.152970 (+ 966us) server_negotiation.cc:407] Connection header received
0808 08:23:23.154510 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.154514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.154583 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.154668 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:23.156022 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.156578 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.157752 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.158111 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.158236 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.161130 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:23.161133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:23.161135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:23.161162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:23.163196 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.163749 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.163756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.163761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.163867 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.164328 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.164335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.164340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.164638 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:23.164771 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.165306 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.165491 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260808 08:23:23.166482 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.152243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:37459)
0808 08:23:23.152781 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.152828 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:23.154278 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:23.154855 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:23.154861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:23.155301 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:23.155847 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.155861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.156727 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.156730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.157613 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.157623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.158285 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.158289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.158567 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.159225 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:23.159242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:23.160907 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:23.163345 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:23.163350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:23.163364 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:23.163636 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:23.163971 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:23.163973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:23.163975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:23.164138 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:23.164757 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:23.164763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:23.165037 (+ 274us) client_negotiation.cc:769] Sending connection context
0808 08:23:23.165206 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 08:23:23.165482 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":122,"threads_started":1}
I20260808 08:23:23.167840 4693 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46083
I20260808 08:23:23.168185 4693 heartbeater.cc:464] Registering TS with master...
I20260808 08:23:23.168951 4693 heartbeater.cc:511] Master 127.12.13.62:46083 requested a full tablet report, sending...
I20260808 08:23:23.170490 4220 ts_manager.cc:194] Registered new tserver with Master: 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
I20260808 08:23:23.171169 4220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:37459
W20260808 08:23:23.298044 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:23.298352 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:23.298403 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:23.302527 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:23.302592 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:23.302654 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:23:23.308151 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:33199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=34809
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:23.309342 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:23.310650 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:23.321249 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:23.321308 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:23.321471 4696 server_base.cc:1034] running on GCE node
W20260808 08:23:23.321309 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:23.322124 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:23.322890 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:23.324126 4696 hybrid_clock.cc:648] HybridClock initialized: now 1786177403324109 us; error 32 us; skew 500 ppm
Aug 08 08:23:23 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177403, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:23.327571 4696 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:23:23.328542 4696 webserver.cc:460] Webserver started at http://127.12.13.3:34809/ using document root <none> and password file <none>
I20260808 08:23:23.329135 4696 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:23.329221 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:23.334101 4696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 08:23:23.336400 4721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:23.338060 4696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:23.338203 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "922b6a3ea0674f8f905b47f4f689703d"
format_stamp: "Formatted at 2026-08-08 08:23:02 on dist-test-slave-6ntk"
server_key: "99f248287c7bc6fc16f3c07407e52cac"
server_key_iv: "55613d57f5bd860ca115a1abd479a89b"
server_key_version: "encryptionkey@0"
I20260808 08:23:23.338793 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:23.376888 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:23.380744 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:23.380918 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:23.381556 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:23.382557 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:23.382611 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:23.382656 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:23.382682 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:23.393023 4696 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:33199
I20260808 08:23:23.393122 4882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:33199 every 8 connection(s)
I20260808 08:23:23.394452 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:23:23.398761 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 4696
Aug 08 08:23:23 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177403, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:23:23.408408 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.399570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.12.13.62:46083)
0808 08:23:23.399924 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.399930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.399945 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:23:23.400084 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.400087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.400136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.400230 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:23.401124 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.401687 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.402643 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.402887 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.402993 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.404474 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:23.404483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:23.404487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:23.404520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:23.406369 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.406804 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.406808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.406810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.406898 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.407242 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.407246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.407248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.407550 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:23.407884 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.407983 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.408167 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"spinlock_wait_cycles":3328,"thread_start_us":114,"threads_started":1}
I20260808 08:23:23.411355 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.396860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:32911 (local address 127.12.13.62:46083)
0808 08:23:23.397035 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.397039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.398024 (+ 985us) server_negotiation.cc:407] Connection header received
0808 08:23:23.400031 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.400034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.400101 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.400221 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:23.401651 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.402325 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.403613 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.403864 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.403988 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.406841 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:23.406844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:23.406846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:23.406872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:23.408646 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.409349 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.409354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.409356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.409453 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.409991 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.409997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.409999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.410206 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:23.410332 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.411041 (+ 709us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.411176 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 08:23:23.412355 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.397285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:32911)
0808 08:23:23.397808 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.397857 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:23.399791 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:23.400426 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:23.400434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:23.400929 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:23.401489 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.401504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.402491 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.402494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.403466 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.403477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.404036 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.404041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.404239 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.404866 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:23.404883 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:23.406645 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:23.408779 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:23.408786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:23.408803 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:23.409229 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:23.409607 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:23.409611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:23.409612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:23.409838 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:23.410334 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:23.410343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:23.410737 (+ 394us) client_negotiation.cc:769] Sending connection context
0808 08:23:23.410948 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 08:23:23.411324 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260808 08:23:23.413595 4884 heartbeater.cc:347] Connected to a master server at 127.12.13.62:46083
I20260808 08:23:23.413928 4884 heartbeater.cc:464] Registering TS with master...
I20260808 08:23:23.414487 4884 heartbeater.cc:511] Master 127.12.13.62:46083 requested a full tablet report, sending...
I20260808 08:23:23.415599 4220 ts_manager.cc:194] Registered new tserver with Master: 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199)
I20260808 08:23:23.416247 4220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:32911
I20260808 08:23:23.422673 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:23:23.427310 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.423406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.12.13.62:46083)
0808 08:23:23.423565 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.423573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.423844 (+ 271us) server_negotiation.cc:407] Connection header received
0808 08:23:23.424008 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.424010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.424123 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.424218 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:23:23.424597 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.425117 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.426212 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.426453 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.426537 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.426850 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:23:23.426992 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 08:23:23.427205 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47576: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 08:23:23.427407 4494 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47576: token expired
W20260808 08:23:23.427876 4120 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.13.62:46083)
I20260808 08:23:23.435846 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.428290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.12.13.62:46083)
0808 08:23:23.428414 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.428418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.428502 (+ 84us) server_negotiation.cc:407] Connection header received
0808 08:23:23.428649 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.428652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.428700 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.428813 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:23.429465 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.430043 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.430848 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.431069 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.431180 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.432533 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:23.432537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:23.432539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:23.432567 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:23.434071 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.434549 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.434553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.434555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.434639 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:23.434928 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:23.434931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:23.434933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:23.435153 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:23.435307 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.435551 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.435670 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 08:23:23.438918 4220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:23:23.442520 4220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:23:23.451793 4394 tablet_service.cc:1467] Processing CreateTablet for tablet b34bd1755cc5408b8468044be461e36c (DEFAULT_TABLE table=test-table [id=406e47acb5b9467ea6f286d5edb8a0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:23:23.452611 4394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34bd1755cc5408b8468044be461e36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:23.455830 4336 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap starting.
I20260808 08:23:23.456172 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.450255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:33199 (local address 127.0.0.1:44594)
0808 08:23:23.450392 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.450410 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:23.450500 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:23.451758 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:23.451761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:23.451781 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:23.451934 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.451945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.453394 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.453398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.454419 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.454429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.455739 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.455742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.455826 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.455851 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 08:23:23.455917 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 08:23:23.456005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":33}
I20260808 08:23:23.456239 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.450694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.12.13.3:33199)
0808 08:23:23.451251 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.451264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.451299 (+ 35us) server_negotiation.cc:407] Connection header received
0808 08:23:23.451387 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.451392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.451586 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.451946 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:23.452175 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.453270 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.454546 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.455507 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.455748 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.455935 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.455989 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.456087 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":130,"threads_started":1}
I20260808 08:23:23.456597 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.450542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:37073 (local address 127.0.0.1:39990)
0808 08:23:23.450893 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.450908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:23.450997 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:23.451714 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:23.451717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:23.451736 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:23.452209 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.452223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.453695 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.453698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.454846 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.454858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.456163 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.456166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.456259 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.456296 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 08:23:23.456382 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 08:23:23.456457 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":2,"thread_start_us":172,"threads_started":1}
I20260808 08:23:23.456938 4336 tablet_bootstrap.cc:654] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:23.457082 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.450763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.12.13.2:37073)
0808 08:23:23.451238 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.451251 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.451282 (+ 31us) server_negotiation.cc:407] Connection header received
0808 08:23:23.451361 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.451365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.451547 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.451744 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:23.452352 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.453566 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.454998 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.455839 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.455948 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.456094 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.456777 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.456904 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":112,"threads_started":1}
I20260808 08:23:23.458923 4336 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: No bootstrap required, opened a new log
I20260808 08:23:23.459024 4336 ts_tablet_manager.cc:1397] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 08:23:23.459482 4336 raft_consensus.cc:348] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.459630 4336 raft_consensus.cc:374] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:23.459674 4336 raft_consensus.cc:729] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8bf840ed95407cae38f025aa8c0c4a, State: Initialized, Role: FOLLOWER
I20260808 08:23:23.459811 4336 consensus_queue.cc:260] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.460091 4336 ts_tablet_manager.cc:1428] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 08:23:23.460317 4794 tablet_service.cc:1467] Processing CreateTablet for tablet b34bd1755cc5408b8468044be461e36c (DEFAULT_TABLE table=test-table [id=406e47acb5b9467ea6f286d5edb8a0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:23:23.460579 4600 tablet_service.cc:1467] Processing CreateTablet for tablet b34bd1755cc5408b8468044be461e36c (DEFAULT_TABLE table=test-table [id=406e47acb5b9467ea6f286d5edb8a0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:23:23.461369 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34bd1755cc5408b8468044be461e36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:23.461731 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34bd1755cc5408b8468044be461e36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:23:23.468740 4904 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Bootstrap starting.
I20260808 08:23:23.468371 4903 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Bootstrap starting.
I20260808 08:23:23.471309 4903 tablet_bootstrap.cc:654] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:23.471678 4904 tablet_bootstrap.cc:654] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Neither blocks nor log segments found. Creating new log.
I20260808 08:23:23.472435 4904 log.cc:824] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:23.472436 4903 log.cc:824] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:23.474591 4903 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: No bootstrap required, opened a new log
I20260808 08:23:23.474674 4904 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: No bootstrap required, opened a new log
I20260808 08:23:23.474819 4903 ts_tablet_manager.cc:1397] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 08:23:23.474941 4904 ts_tablet_manager.cc:1397] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:23:23.478026 4904 raft_consensus.cc:348] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.478065 4903 raft_consensus.cc:348] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.478308 4904 raft_consensus.cc:374] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:23.478302 4903 raft_consensus.cc:374] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:23:23.478379 4904 raft_consensus.cc:729] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f19dc4aa4594e019378967a7cbb4df3, State: Initialized, Role: FOLLOWER
I20260808 08:23:23.478375 4903 raft_consensus.cc:729] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922b6a3ea0674f8f905b47f4f689703d, State: Initialized, Role: FOLLOWER
I20260808 08:23:23.478823 4903 consensus_queue.cc:260] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.478845 4904 consensus_queue.cc:260] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.479527 4693 heartbeater.cc:503] Master 127.12.13.62:46083 was elected leader, sending a full tablet report...
I20260808 08:23:23.479527 4884 heartbeater.cc:503] Master 127.12.13.62:46083 was elected leader, sending a full tablet report...
I20260808 08:23:23.479928 4904 ts_tablet_manager.cc:1428] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 08:23:23.479929 4903 ts_tablet_manager.cc:1428] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260808 08:23:23.645633 4886 tablet.cc:2369] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:23:23.651101 4694 tablet.cc:2369] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:23:23.661557 4499 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:23.661844 4499 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.663390 4499 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
I20260808 08:23:23.667641 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.663725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:33199 (local address 127.12.13.1:55097)
0808 08:23:23.664043 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.664060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:23.664185 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:23.664591 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:23.664594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:23.664612 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:23.664704 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.664708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.665466 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.665471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.666368 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.666378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.667131 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.667135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.667230 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.667260 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 08:23:23.667371 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 08:23:23.667451 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":144,"threads_started":1}
I20260808 08:23:23.667627 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.663849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:55097 (local address 127.12.13.3:33199)
0808 08:23:23.664005 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.664010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.664215 (+ 205us) server_negotiation.cc:407] Connection header received
0808 08:23:23.664322 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.664325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.664384 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.664467 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:23.664831 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.665343 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.666503 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.666997 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.667128 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.667201 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.667414 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.667491 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:23:23.667847 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.664182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:37073 (local address 127.12.13.1:40665)
0808 08:23:23.664566 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.664579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:23.664704 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:23.664944 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:23.664946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:23.664963 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:23:23.665082 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.665087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.665820 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.665825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.666624 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:23.666633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.667422 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.667426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:23.667532 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.667558 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 08:23:23.667640 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 08:23:23.667718 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":84,"threads_started":1}
I20260808 08:23:23.667910 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.664360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:40665 (local address 127.12.13.2:37073)
0808 08:23:23.664523 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.664527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.664661 (+ 134us) server_negotiation.cc:407] Connection header received
0808 08:23:23.664806 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.664808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.664848 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.664933 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:23:23.665190 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.665693 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.666753 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.667295 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.667396 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.667456 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.667676 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.667766 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:23:23.668365 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34bd1755cc5408b8468044be461e36c" candidate_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922b6a3ea0674f8f905b47f4f689703d" is_pre_election: true
I20260808 08:23:23.668691 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34bd1755cc5408b8468044be461e36c" candidate_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f19dc4aa4594e019378967a7cbb4df3" is_pre_election: true
I20260808 08:23:23.668740 4818 raft_consensus.cc:2454] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f8bf840ed95407cae38f025aa8c0c4a in term 0.
I20260808 08:23:23.668992 4628 raft_consensus.cc:2454] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f8bf840ed95407cae38f025aa8c0c4a in term 0.
I20260808 08:23:23.669317 4330 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a, 922b6a3ea0674f8f905b47f4f689703d; no voters:
I20260808 08:23:23.669653 4499 raft_consensus.cc:2781] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:23:23.669740 4499 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:23:23.669777 4499 raft_consensus.cc:3037] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:23.670949 4499 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.671331 4499 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 1 election: Requested vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
I20260808 08:23:23.671814 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34bd1755cc5408b8468044be461e36c" candidate_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922b6a3ea0674f8f905b47f4f689703d"
I20260808 08:23:23.671904 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34bd1755cc5408b8468044be461e36c" candidate_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f19dc4aa4594e019378967a7cbb4df3"
I20260808 08:23:23.671977 4818 raft_consensus.cc:3037] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:23.672065 4628 raft_consensus.cc:3037] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:23:23.673146 4628 raft_consensus.cc:2454] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8bf840ed95407cae38f025aa8c0c4a in term 1.
I20260808 08:23:23.673179 4818 raft_consensus.cc:2454] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8bf840ed95407cae38f025aa8c0c4a in term 1.
I20260808 08:23:23.673501 4330 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a, 7f19dc4aa4594e019378967a7cbb4df3; no voters:
I20260808 08:23:23.673782 4499 raft_consensus.cc:2781] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:23:23.674043 4499 raft_consensus.cc:686] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 LEADER]: Becoming Leader. State: Replica: 1f8bf840ed95407cae38f025aa8c0c4a, State: Running, Role: LEADER
I20260808 08:23:23.674342 4499 consensus_queue.cc:237] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:23.677471 4220 catalog_manager.cc:5495] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1). New cstate: current_term: 1 leader_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } health_report { overall_health: HEALTHY } } }
W20260808 08:23:23.685842 4493 tablet.cc:2369] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:23:23.697494 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.693245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.12.13.1:46109)
0808 08:23:23.693584 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:23.693594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:23.693609 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:23:23.693696 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:23.693699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:23.693765 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:23.693944 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:23:23.694286 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.695129 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.696080 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:23.696314 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:23.696411 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:23.696656 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:23:23.697121 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:23:23.697214 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:23.697289 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 08:23:23.697349 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":136,"threads_started":1}
I20260808 08:23:23.701939 4818 raft_consensus.cc:1261] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Refusing update from remote peer 1f8bf840ed95407cae38f025aa8c0c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:23:23.701952 4628 raft_consensus.cc:1261] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Refusing update from remote peer 1f8bf840ed95407cae38f025aa8c0c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:23:23.702788 4912 consensus_queue.cc:1035] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:23:23.703012 4499 consensus_queue.cc:1035] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:23:23.709694 4918 mvcc.cc:205] Tried to move back new op lower bound from 7316182645554970624 to 7316182645452664832. Current Snapshot: MvccSnapshot[applied={T|T < 7316182645554970624}]
I20260808 08:23:23.710439 4920 mvcc.cc:205] Tried to move back new op lower bound from 7316182645554970624 to 7316182645452664832. Current Snapshot: MvccSnapshot[applied={T|T < 7316182645554970624}]
I20260808 08:23:23.720634 4220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:23:23.720850 4220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:23:23.723496 4220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Sending DeleteTablet for 3 replicas of tablet b34bd1755cc5408b8468044be461e36c
I20260808 08:23:23.724336 4394 tablet_service.cc:1514] Processing DeleteTablet for tablet b34bd1755cc5408b8468044be461e36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:23 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:34152
I20260808 08:23:23.724395 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 4273
I20260808 08:23:23.724362 4794 tablet_service.cc:1514] Processing DeleteTablet for tablet b34bd1755cc5408b8468044be461e36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:23 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:44594
I20260808 08:23:23.731012 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet b34bd1755cc5408b8468044be461e36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:23:23 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:39990
I20260808 08:23:23.731678 4929 tablet_replica.cc:330] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: stopping tablet replica
I20260808 08:23:23.732240 4929 raft_consensus.cc:2229] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:23:23.732692 4929 raft_consensus.cc:2258] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:23:23.732785 4927 tablet_replica.cc:330] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: stopping tablet replica
I20260808 08:23:23.733161 4927 raft_consensus.cc:2229] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:23:23.733511 4927 raft_consensus.cc:2258] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:23:23.734671 4929 ts_tablet_manager.cc:1905] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:23:23.734992 4927 ts_tablet_manager.cc:1905] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:23:23.737473 4927 ts_tablet_manager.cc:1918] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:23:23.737599 4927 log.cc:1196] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/b34bd1755cc5408b8468044be461e36c
I20260808 08:23:23.738060 4927 ts_tablet_manager.cc:1939] T b34bd1755cc5408b8468044be461e36c P 922b6a3ea0674f8f905b47f4f689703d: Deleting consensus metadata
I20260808 08:23:23.738617 4929 ts_tablet_manager.cc:1918] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:23:23.738731 4929 log.cc:1196] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/b34bd1755cc5408b8468044be461e36c
I20260808 08:23:23.739082 4929 ts_tablet_manager.cc:1939] T b34bd1755cc5408b8468044be461e36c P 7f19dc4aa4594e019378967a7cbb4df3: Deleting consensus metadata
I20260808 08:23:23.739522 4201 catalog_manager.cc:4841] TS 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): tablet b34bd1755cc5408b8468044be461e36c (table test-table [id=406e47acb5b9467ea6f286d5edb8a0db]) successfully deleted
I20260808 08:23:23.740442 4201 catalog_manager.cc:4841] TS 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): tablet b34bd1755cc5408b8468044be461e36c (table test-table [id=406e47acb5b9467ea6f286d5edb8a0db]) successfully deleted
W20260808 08:23:23.741390 4201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:46109 (error 108)
I20260808 08:23:23.742419 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 4496
I20260808 08:23:23.742633 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:23.742260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:46109 (local address 127.0.0.1:34156)
0808 08:23:23.742438 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:23.742527 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 08:23:23.749692 4201 catalog_manager.cc:4584] TS 1f8bf840ed95407cae38f025aa8c0c4a (127.12.13.1:46109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b34bd1755cc5408b8468044be461e36c: Network error: Client connection negotiation failed: client connection to 127.12.13.1:46109: connect: Connection refused (error 111)
I20260808 08:23:23.762671 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 4696
I20260808 08:23:23.774758 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 4172
I20260808 08:23:43.790452 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46083
--webserver_interface=127.12.13.62
--webserver_port=45379
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:23:43.913080 4931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:43.913400 4931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:43.913443 4931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:43.917557 4931 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 08:23:43.917644 4931 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 08:23:43.917660 4931 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 08:23:43.917675 4931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:23:43.917694 4931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:43.917712 4931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:43.917729 4931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:23:43.917744 4931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:23:43.923533 4931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:43.924803 4931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:43.925947 4931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:43.936162 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:43.936178 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:43.936163 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:43.936568 4931 server_base.cc:1034] running on GCE node
I20260808 08:23:43.937240 4931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:43.938617 4931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:43.939841 4931 hybrid_clock.cc:648] HybridClock initialized: now 1786177423939805 us; error 52 us; skew 500 ppm
Aug 08 08:23:43 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177423, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:43.948067 4931 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:23:43.949333 4931 webserver.cc:460] Webserver started at http://127.12.13.62:45379/ using document root <none> and password file <none>
I20260808 08:23:43.950033 4931 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:43.950133 4931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:43.954349 4931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:43.956796 4953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:43.958242 4931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:23:43.958433 4931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "847f456a548c492ba978c4735fae56c7"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "a474a02080453a8bbae5f25462581353"
server_key_iv: "86ece7d03aa36ac009b815c730a3fdd3"
server_key_version: "encryptionkey@0"
I20260808 08:23:43.959030 4931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:43.974113 4931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:43.977833 4931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:43.978093 4931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:43.985745 4931 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:46083
I20260808 08:23:43.986130 5036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:46083 every 8 connection(s)
I20260808 08:23:43.987104 4931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:23:43.989722 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 4931
I20260808 08:23:43.990262 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:46109
--local_ip_for_outbound_sockets=127.12.13.1
--tserver_master_addrs=127.12.13.62:46083
--webserver_port=38917
--webserver_interface=127.12.13.1
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:23:43.992606 5038 sys_catalog.cc:263] Verifying existing consensus state
I20260808 08:23:43.994576 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap starting.
I20260808 08:23:44.013569 5038 log.cc:824] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:44.025367 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:23:44.025795 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Bootstrap complete.
I20260808 08:23:44.030176 5038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:44.030614 5038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847f456a548c492ba978c4735fae56c7, State: Initialized, Role: FOLLOWER
I20260808 08:23:44.031127 5038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:44.031291 5038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:23:44.031378 5038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:23:44.031502 5038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 2 FOLLOWER]: Advancing to term 3
I20260808 08:23:44.033475 5038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:44.033829 5038 leader_election.cc:302] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 847f456a548c492ba978c4735fae56c7; no voters:
I20260808 08:23:44.034262 5038 leader_election.cc:288] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 08:23:44.034361 5044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 08:23:44.034608 5044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [term 3 LEADER]: Becoming Leader. State: Replica: 847f456a548c492ba978c4735fae56c7, State: Running, Role: LEADER
I20260808 08:23:44.034934 5044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } }
I20260808 08:23:44.035597 5038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:23:44.036458 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "847f456a548c492ba978c4735fae56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } } }
I20260808 08:23:44.036616 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: This master's current role is: LEADER
I20260808 08:23:44.037086 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 847f456a548c492ba978c4735fae56c7. Latest consensus state: current_term: 3 leader_uuid: "847f456a548c492ba978c4735fae56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847f456a548c492ba978c4735fae56c7" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 46083 } } }
I20260808 08:23:44.037268 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7 [sys.catalog]: This master's current role is: LEADER
I20260808 08:23:44.037684 5055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:23:44.043120 5055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:23:44.043813 5055 catalog_manager.cc:1259] Loaded cluster ID: 50199b3ef9c9411c8f310f35370b50f1
I20260808 08:23:44.043876 5055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:23:44.046551 5055 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:23:44.047384 5055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Loaded TSK: 0
I20260808 08:23:44.047470 5055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 847f456a548c492ba978c4735fae56c7: Loaded TSK: 1
F20260808 08:23:44.048720 5055 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd525e5a9ad google::LogMessage::Fail() at ??:0
@ 0x7fd525e5d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd525e5a4de google::LogMessage::Flush() at ??:0
@ 0x7fd525e5ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd528b1e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd528b16f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd5307c37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd53079d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd53079e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd5307dfdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd5307fe82b std::function<>::operator()() at ??:0
@ 0x7fd53079db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd53079ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd53079b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd5307df640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd5280ac3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd5280acc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd5280ae3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd52809d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd5296e16db start_thread at ??:0
@ 0x7fd52495861f clone at ??:0
W20260808 08:23:44.151901 5041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:44.152175 5041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:44.152222 5041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:44.156606 5041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:44.156673 5041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:44.156777 5041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:23:44.163154 5041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:46109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=38917
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:44.164389 5041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:44.165512 5041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:44.175830 5103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:44.175820 5104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:44.176211 5041 server_base.cc:1034] running on GCE node
W20260808 08:23:44.175843 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:44.178759 5041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:44.179472 5041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:44.180663 5041 hybrid_clock.cc:648] HybridClock initialized: now 1786177424180632 us; error 53 us; skew 500 ppm
Aug 08 08:23:44 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177424, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:44.183827 5041 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:23:44.184815 5041 webserver.cc:460] Webserver started at http://127.12.13.1:38917/ using document root <none> and password file <none>
I20260808 08:23:44.185566 5041 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:44.185683 5041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:44.192173 5041 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:23:44.195827 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.197346 5041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:23:44.197500 5041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "1f8bf840ed95407cae38f025aa8c0c4a"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "d1676f569bfcb8edb95fe70bca61c269"
server_key_iv: "06ec1683a8b933abcad7f6e1bbd95c86"
server_key_version: "encryptionkey@0"
I20260808 08:23:44.198143 5041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:44.211899 5041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:44.215587 5041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:44.215819 5041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:44.216472 5041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:44.217675 5131 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 08:23:44.223568 5041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 08:23:44.223654 5041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 08:23:44.223743 5041 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 08:23:44.225541 5041 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 08:23:44.225616 5041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 08:23:44.225993 5131 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap starting.
I20260808 08:23:44.237035 5041 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:46109
I20260808 08:23:44.237287 5271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:46109 every 8 connection(s)
I20260808 08:23:44.238219 5041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:23:44.241353 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 5041
I20260808 08:23:44.241823 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:37073
--local_ip_for_outbound_sockets=127.12.13.2
--tserver_master_addrs=127.12.13.62:46083
--webserver_port=35575
--webserver_interface=127.12.13.2
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:23:44.250204 5131 log.cc:824] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:23:44.260141 5131 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 08:23:44.260632 5131 tablet_bootstrap.cc:492] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Bootstrap complete.
I20260808 08:23:44.262408 5131 ts_tablet_manager.cc:1397] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.010s
I20260808 08:23:44.266103 5131 raft_consensus.cc:348] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:44.266687 5131 raft_consensus.cc:729] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8bf840ed95407cae38f025aa8c0c4a, State: Initialized, Role: FOLLOWER
I20260808 08:23:44.267270 5131 consensus_queue.cc:260] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:44.268391 5131 ts_tablet_manager.cc:1428] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260808 08:23:44.311543 5279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:23:44.240319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:35499)
0808 08:23:44.240834 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.240892 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:44.244837 (+ 3945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:44.310717 (+ 65880us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":156,"threads_started":1}
W20260808 08:23:44.312083 5274 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:46083 (0 consecutive failures): Network error: Failed to ping master at 127.12.13.62:46083: Client connection negotiation failed: client connection to 127.12.13.62:46083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 08:23:44.312911 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.312648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:51587)
0808 08:23:44.312785 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.312852 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 08:23:44.313593 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.313372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:55767)
0808 08:23:44.313483 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.313534 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:23:44.313771 5274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:23:44.366695 5281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:44.367026 5281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:44.367102 5281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:44.371310 5281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:44.371405 5281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:44.371500 5281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:23:44.376868 5281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:37073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=35575
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:44.378068 5281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:44.379169 5281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:44.389573 5294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:44.389585 5297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:44.389691 5281 server_base.cc:1034] running on GCE node
W20260808 08:23:44.389586 5301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:44.390319 5281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:44.391010 5281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:44.392179 5281 hybrid_clock.cc:648] HybridClock initialized: now 1786177424392145 us; error 46 us; skew 500 ppm
Aug 08 08:23:44 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177424, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:44.395174 5281 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:23:44.396086 5281 webserver.cc:460] Webserver started at http://127.12.13.2:35575/ using document root <none> and password file <none>
I20260808 08:23:44.396687 5281 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:44.396751 5281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:44.401443 5281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:23:44.403846 5311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.405164 5281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:23:44.405310 5281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "7f19dc4aa4594e019378967a7cbb4df3"
format_stamp: "Formatted at 2026-08-08 08:23:01 on dist-test-slave-6ntk"
server_key: "1959be2ec989b4b58dec5f72f9912313"
server_key_iv: "914308c90dfe641fb048e85ca5b0c3c9"
server_key_version: "encryptionkey@0"
I20260808 08:23:44.405953 5281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:44.443670 5281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:44.447362 5281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:44.447594 5281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:44.448275 5281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:44.449182 5281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:44.449256 5281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.449334 5281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:44.449378 5281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.458953 5281 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:37073
I20260808 08:23:44.458978 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:37073 every 8 connection(s)
I20260808 08:23:44.460035 5281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:23:44.461001 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 5281
I20260808 08:23:44.461498 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:33199
--local_ip_for_outbound_sockets=127.12.13.3
--tserver_master_addrs=127.12.13.62:46083
--webserver_port=34809
--webserver_interface=127.12.13.3
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:23:44.464443 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.462531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:33603)
0808 08:23:44.463040 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.463172 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":144,"threads_started":1}
W20260808 08:23:44.465063 5459 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:46083 (0 consecutive failures): Network error: Failed to ping master at 127.12.13.62:46083: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
I20260808 08:23:44.465770 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.465528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:37083)
0808 08:23:44.465656 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.465714 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 08:23:44.466564 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.466355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:47581)
0808 08:23:44.466470 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.466511 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 08:23:44.466826 5459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 08:23:44.584903 5464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:23:44.585273 5464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:23:44.585438 5464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:23:44.589628 5464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:23:44.589725 5464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:23:44.589823 5464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:23:44.595234 5464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:33199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=34809
--tserver_master_addrs=127.12.13.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:23:44.596884 5464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:23:44.598119 5464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:23:44.609313 5471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:44.609556 5464 server_base.cc:1034] running on GCE node
W20260808 08:23:44.609351 5470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:23:44.609331 5473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:23:44.610232 5464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:23:44.610973 5464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:23:44.612176 5464 hybrid_clock.cc:648] HybridClock initialized: now 1786177424612155 us; error 34 us; skew 500 ppm
Aug 08 08:23:44 dist-test-slave-6ntk krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177424, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:23:44.615274 5464 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:23:44.616276 5464 webserver.cc:460] Webserver started at http://127.12.13.3:34809/ using document root <none> and password file <none>
I20260808 08:23:44.616868 5464 fs_manager.cc:359] Metadata directory not provided
I20260808 08:23:44.616930 5464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:23:44.621212 5464 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:23:44.623596 5484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.624971 5464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:23:44.625110 5464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "922b6a3ea0674f8f905b47f4f689703d"
format_stamp: "Formatted at 2026-08-08 08:23:02 on dist-test-slave-6ntk"
server_key: "99f248287c7bc6fc16f3c07407e52cac"
server_key_iv: "55613d57f5bd860ca115a1abd479a89b"
server_key_version: "encryptionkey@0"
I20260808 08:23:44.625694 5464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:23:44.655293 5464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:23:44.658687 5464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:23:44.658919 5464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:23:44.659643 5464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:23:44.660602 5464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:23:44.660672 5464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.660745 5464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:23:44.660789 5464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:23:44.671000 5464 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:33199
I20260808 08:23:44.671023 5649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:33199 every 8 connection(s)
I20260808 08:23:44.672097 5464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:23:44.676663 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.674582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:50931)
0808 08:23:44.675124 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.675305 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":125,"threads_started":1}
W20260808 08:23:44.677155 5650 heartbeater.cc:644] Failed to heartbeat to 127.12.13.62:46083 (0 consecutive failures): Network error: Failed to ping master at 127.12.13.62:46083: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
I20260808 08:23:44.677933 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.677648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:48581)
0808 08:23:44.677772 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.677823 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 08:23:44.678612 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:44.678410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:58251)
0808 08:23:44.678516 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:44.678561 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 08:23:44.678813 5650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 08:23:44.681298 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 5464
I20260808 08:23:45.314940 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.314437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:40109)
0808 08:23:45.314748 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:45.314816 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260808 08:23:45.467890 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.467453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:57043)
0808 08:23:45.467711 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:45.467771 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260808 08:23:45.679953 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.679471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:50645)
0808 08:23:45.679772 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:45.679838 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260808 08:23:45.845418 5659 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:45.845690 5659 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:45.846998 5659 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
Aug 08 08:23:45 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177424, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.3@KRBTEST.COM
Aug 08 08:23:45 dist-test-slave-6ntk krb5kdc[3351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177424, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.2@KRBTEST.COM
I20260808 08:23:45.862421 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.847236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:33199 (local address 127.12.13.1:38959)
0808 08:23:45.847643 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:45.847659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:45.847866 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:45.849811 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:45.849817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:45.850450 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:45.850928 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:45.850944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.852259 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.852262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:45.853315 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:45.853332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.854127 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.854131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:45.854354 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:45.855036 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:45.855052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:45.856765 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:45.859430 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:45.859435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:45.859446 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:45.859719 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:45.860138 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:45.860140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:45.860142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:45.860335 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:45.861192 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:45.861199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:45.861558 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 08:23:45.861786 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 08:23:45.862200 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":235,"spinlock_wait_cycles":62336,"thread_start_us":102,"threads_started":1}
I20260808 08:23:45.862463 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.847990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:45539 (local address 127.12.13.2:37073)
0808 08:23:45.848406 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:45.848425 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:45.848456 (+ 31us) server_negotiation.cc:407] Connection header received
0808 08:23:45.849192 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:45.849202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:45.849597 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:45.849918 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:45.851115 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.852090 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.853495 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.854009 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.854194 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:45.857276 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:45.857282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:45.857296 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:45.857332 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:45.859310 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:45.860105 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:45.860113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:45.860117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:45.860296 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:45.860620 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:45.860623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:45.860626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:45.861064 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:45.861182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:45.861958 (+ 776us) server_negotiation.cc:299] Negotiation successful
0808 08:23:45.862265 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":90,"threads_started":1}
I20260808 08:23:45.862421 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.847768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:37073 (local address 127.12.13.1:45539)
0808 08:23:45.848083 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:45.848099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:23:45.848192 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:23:45.849860 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:23:45.849863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:23:45.850449 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:23:45.850929 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:45.850944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.852261 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.852264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:45.853354 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:23:45.853364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.854149 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.854153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:23:45.854352 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:45.855097 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:23:45.855110 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:23:45.857079 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:23:45.859808 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:45.859810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:45.859812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:45.859966 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:45.860406 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:23:45.860409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:23:45.860412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:23:45.860505 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:23:45.861226 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:23:45.861229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:23:45.861558 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 08:23:45.861786 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 08:23:45.862257 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":295,"spinlock_wait_cycles":7936,"thread_start_us":80,"threads_started":1}
I20260808 08:23:45.862463 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:45.847480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:38959 (local address 127.12.13.3:33199)
0808 08:23:45.847959 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 08:23:45.847993 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0808 08:23:45.848052 (+ 59us) server_negotiation.cc:407] Connection header received
0808 08:23:45.848907 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:23:45.848916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:23:45.849354 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:23:45.849709 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:23:45.851118 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.852103 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.853446 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:23:45.854003 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:23:45.854171 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:23:45.856991 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:23:45.856996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:23:45.857008 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:23:45.857037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:23:45.859310 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:45.859838 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:45.859844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:45.859849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:45.860007 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:23:45.860454 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:23:45.860461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:23:45.860464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:23:45.861027 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:23:45.861182 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 08:23:45.861958 (+ 776us) server_negotiation.cc:299] Negotiation successful
0808 08:23:45.862265 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":84,"threads_started":1}
W20260808 08:23:45.865986 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
W20260808 08:23:45.866184 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
I20260808 08:23:45.866230 5122 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a; no voters: 7f19dc4aa4594e019378967a7cbb4df3, 922b6a3ea0674f8f905b47f4f689703d
I20260808 08:23:45.866504 5659 raft_consensus.cc:2726] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:23:46.316013 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:46.315684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:47701)
0808 08:23:46.315857 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:46.315920 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 08:23:46.469393 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:46.468839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:60183)
0808 08:23:46.469191 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:46.469273 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":158,"threads_started":1}
I20260808 08:23:46.681246 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:46.680729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:59541)
0808 08:23:46.681058 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:46.681121 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260808 08:23:47.317394 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:47.316958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:44495)
0808 08:23:47.317224 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:47.317290 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260808 08:23:47.470801 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:47.470313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:55471)
0808 08:23:47.470632 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:47.470699 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
I20260808 08:23:47.682788 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:47.682260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:58529)
0808 08:23:47.682594 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:47.682662 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260808 08:23:47.719682 5672 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:47.719888 5672 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:47.720418 5672 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
W20260808 08:23:47.721581 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
W20260808 08:23:47.721764 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
I20260808 08:23:47.721834 5122 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a; no voters: 7f19dc4aa4594e019378967a7cbb4df3, 922b6a3ea0674f8f905b47f4f689703d
I20260808 08:23:47.722115 5672 raft_consensus.cc:2726] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:23:48.318775 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:48.318254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:35069)
0808 08:23:48.318562 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:48.318629 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260808 08:23:48.472077 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:48.471630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:60397)
0808 08:23:48.471921 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:48.471976 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
I20260808 08:23:48.684046 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:48.683549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:59765)
0808 08:23:48.683888 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:48.683947 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
I20260808 08:23:49.320366 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:49.319857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:43561)
0808 08:23:49.320174 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:49.320234 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":85,"threads_started":1}
I20260808 08:23:49.473477 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:49.473015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:36957)
0808 08:23:49.473311 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:49.473374 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260808 08:23:49.685559 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:49.685006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:53549)
0808 08:23:49.685380 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:49.685451 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":116,"threads_started":1}
I20260808 08:23:50.215449 5681 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:50.215581 5681 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:50.216002 5681 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
W20260808 08:23:50.216956 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
W20260808 08:23:50.217170 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
I20260808 08:23:50.217355 5122 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a; no voters: 7f19dc4aa4594e019378967a7cbb4df3, 922b6a3ea0674f8f905b47f4f689703d
I20260808 08:23:50.217549 5681 raft_consensus.cc:2726] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:23:50.321863 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:50.321332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:58565)
0808 08:23:50.321678 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:50.321751 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
I20260808 08:23:50.474902 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:50.474384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:50359)
0808 08:23:50.474699 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:50.474762 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260808 08:23:50.686935 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:50.686432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:45161)
0808 08:23:50.686727 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:50.686792 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260808 08:23:51.323138 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:51.322723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:50037)
0808 08:23:51.322974 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:51.323036 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260808 08:23:51.476413 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:51.475836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:54837)
0808 08:23:51.476248 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:51.476310 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":92,"threads_started":1}
I20260808 08:23:51.688421 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:51.687882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:56279)
0808 08:23:51.688231 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:51.688320 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":82,"threads_started":1}
I20260808 08:23:52.240130 5688 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:52.240310 5688 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:52.240701 5688 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
W20260808 08:23:52.241652 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
W20260808 08:23:52.241832 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
I20260808 08:23:52.241943 5122 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a; no voters: 7f19dc4aa4594e019378967a7cbb4df3, 922b6a3ea0674f8f905b47f4f689703d
I20260808 08:23:52.242115 5688 raft_consensus.cc:2726] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:23:52.324662 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:52.324095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:59001)
0808 08:23:52.324491 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:52.324558 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":213,"threads_started":1}
I20260808 08:23:52.477851 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:52.477360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:48877)
0808 08:23:52.477677 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:52.477741 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":116,"threads_started":1}
I20260808 08:23:52.690106 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:52.689463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:50713)
0808 08:23:52.689811 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:52.689983 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":90,"threads_started":1}
I20260808 08:23:53.326140 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:53.325573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:53907)
0808 08:23:53.325946 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:53.326028 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260808 08:23:53.479494 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:53.478926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:51893)
0808 08:23:53.479286 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:53.479389 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260808 08:23:53.691705 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:53.691084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:58903)
0808 08:23:53.691505 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:53.691570 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":185,"threads_started":1}
I20260808 08:23:54.327486 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:54.327061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:57989)
0808 08:23:54.327323 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:54.327385 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260808 08:23:54.481086 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:54.480577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:51453)
0808 08:23:54.480897 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:54.480970 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
I20260808 08:23:54.693030 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:54.692583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:46615)
0808 08:23:54.692869 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:54.692927 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
I20260808 08:23:54.932168 5702 raft_consensus.cc:482] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:23:54.932406 5702 raft_consensus.cc:504] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922b6a3ea0674f8f905b47f4f689703d" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 33199 } } peers { permanent_uuid: "7f19dc4aa4594e019378967a7cbb4df3" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 37073 } } peers { permanent_uuid: "1f8bf840ed95407cae38f025aa8c0c4a" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 46109 } }
I20260808 08:23:54.932919 5702 leader_election.cc:288] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199), 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073)
W20260808 08:23:54.933837 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 922b6a3ea0674f8f905b47f4f689703d (127.12.13.3:33199): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
W20260808 08:23:54.934068 5122 leader_election.cc:341] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f19dc4aa4594e019378967a7cbb4df3 (127.12.13.2:37073): Not found: Tablet not found: b34bd1755cc5408b8468044be461e36c
I20260808 08:23:54.934154 5122 leader_election.cc:302] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f8bf840ed95407cae38f025aa8c0c4a; no voters: 7f19dc4aa4594e019378967a7cbb4df3, 922b6a3ea0674f8f905b47f4f689703d
I20260808 08:23:54.934403 5702 raft_consensus.cc:2726] T b34bd1755cc5408b8468044be461e36c P 1f8bf840ed95407cae38f025aa8c0c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 08:23:55.329044 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:55.328456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:43877)
0808 08:23:55.328812 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:55.328898 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
I20260808 08:23:55.482610 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:55.482112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:60183)
0808 08:23:55.482431 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:55.482504 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260808 08:23:55.694717 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:55.694187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:33733)
0808 08:23:55.694531 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:55.694606 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":97,"threads_started":1}
I20260808 08:23:56.330659 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:56.330120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:43553)
0808 08:23:56.330487 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:56.330552 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":103,"threads_started":1}
I20260808 08:23:56.484102 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:56.483607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:51485)
0808 08:23:56.483928 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:56.483997 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20260808 08:23:56.696417 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:56.695861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:48725)
0808 08:23:56.696230 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:56.696303 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":127,"threads_started":1}
I20260808 08:23:57.332497 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:57.331935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:42245)
0808 08:23:57.332319 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:57.332390 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":93,"threads_started":1}
I20260808 08:23:57.485646 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:57.485140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:47333)
0808 08:23:57.485467 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:57.485538 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260808 08:23:57.698203 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:57.697528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:33327)
0808 08:23:57.697932 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:57.698024 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260808 08:23:58.333921 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:58.333331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:42809)
0808 08:23:58.333663 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:58.333753 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":86,"threads_started":1}
I20260808 08:23:58.487150 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:58.486662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:37715)
0808 08:23:58.486968 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:58.487043 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260808 08:23:58.699615 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:58.699141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:55643)
0808 08:23:58.699458 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:58.699515 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":129,"threads_started":1}
I20260808 08:23:59.335464 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:59.334986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.1:42595)
0808 08:23:59.335268 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:59.335358 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260808 08:23:59.488785 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:59.488297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.2:43971)
0808 08:23:59.488610 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:59.488679 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260808 08:23:59.691532 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 5041
I20260808 08:23:59.701093 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:23:59.700643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:46083 (local address 127.12.13.3:55915)
0808 08:23:59.700917 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 08:23:59.700992 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
I20260808 08:23:59.704568 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 5281
I20260808 08:23:59.718510 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 5464
2026-08-08T08:23:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59876 ms)
[----------] 30 tests from SecurityITest (259830 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:23:59 dist-test-slave-6ntk krb5kdc[5728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:23:59 dist-test-slave-6ntk krb5kdc[5728](info): set up 2 sockets
Aug 08 08:23:59 dist-test-slave-6ntk krb5kdc[5728](info): commencing operation
krb5kdc: starting...
W20260808 08:24:00.994278 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:01.133857 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36659
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:01.256542 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:01.256922 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:01.257015 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:01.261221 5742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:01.261346 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:01.261389 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:01.261435 5742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:01.261476 5742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:01.267247 5742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:01.268499 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:01.269671 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:01.279163 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:01.279211 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:01.279163 5752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:01.279249 5742 server_base.cc:1034] running on GCE node
I20260808 08:24:01.280084 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:01.281308 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:01.282512 5742 hybrid_clock.cc:648] HybridClock initialized: now 1786177441282496 us; error 33 us; skew 500 ppm
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:01.289665 5742 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:01.290787 5742 webserver.cc:460] Webserver started at http://127.12.13.62:42609/ using document root <none> and password file <none>
I20260808 08:24:01.291411 5742 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:01.291507 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:01.291761 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:01.294037 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "7120e60e3f7844319c7be9568ce8833a"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "9fe015611f0ea84f05e6387448f2b4e1"
server_key_iv: "0cf7efa5ff5d82ab0d8b12a64844869c"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.294621 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "7120e60e3f7844319c7be9568ce8833a"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "9fe015611f0ea84f05e6387448f2b4e1"
server_key_iv: "0cf7efa5ff5d82ab0d8b12a64844869c"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.298442 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 08:24:01.300956 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.302258 5742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:24:01.302454 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "7120e60e3f7844319c7be9568ce8833a"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "9fe015611f0ea84f05e6387448f2b4e1"
server_key_iv: "0cf7efa5ff5d82ab0d8b12a64844869c"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.302573 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:01.349632 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:01.354045 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:01.354262 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:01.362357 5742 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36659
I20260808 08:24:01.362387 5834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36659 every 8 connection(s)
I20260808 08:24:01.363461 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:01.366528 5835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:01.372490 5835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: Bootstrap starting.
I20260808 08:24:01.372788 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 5742
I20260808 08:24:01.372953 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:01.373257 12340 external_mini_cluster.cc:1428] Setting key b5ca3f4b352482652fcc125e62d89ecb
I20260808 08:24:01.375375 5835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:01.376288 5835 log.cc:824] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:01.378640 5835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: No bootstrap required, opened a new log
I20260808 08:24:01.382848 5835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7120e60e3f7844319c7be9568ce8833a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36659 } }
I20260808 08:24:01.383143 5835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:01.383209 5835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7120e60e3f7844319c7be9568ce8833a, State: Initialized, Role: FOLLOWER
I20260808 08:24:01.383723 5835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7120e60e3f7844319c7be9568ce8833a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36659 } }
I20260808 08:24:01.383913 5835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:01.383994 5835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:01.384109 5835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:01.385426 5835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7120e60e3f7844319c7be9568ce8833a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36659 } }
I20260808 08:24:01.385828 5835 leader_election.cc:302] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7120e60e3f7844319c7be9568ce8833a; no voters:
I20260808 08:24:01.386256 5835 leader_election.cc:288] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:01.386384 5840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:01.386706 5840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [term 1 LEADER]: Becoming Leader. State: Replica: 7120e60e3f7844319c7be9568ce8833a, State: Running, Role: LEADER
I20260808 08:24:01.387084 5840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7120e60e3f7844319c7be9568ce8833a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36659 } }
I20260808 08:24:01.387728 5835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:01.388836 5841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7120e60e3f7844319c7be9568ce8833a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7120e60e3f7844319c7be9568ce8833a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36659 } } }
I20260808 08:24:01.388974 5841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:01.389420 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7120e60e3f7844319c7be9568ce8833a. Latest consensus state: current_term: 1 leader_uuid: "7120e60e3f7844319c7be9568ce8833a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7120e60e3f7844319c7be9568ce8833a" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36659 } } }
I20260808 08:24:01.389565 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:01.389714 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:01.374746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.12.13.62:36659)
0808 08:24:01.375219 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:01.375241 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:01.375295 (+ 54us) server_negotiation.cc:407] Connection header received
0808 08:24:01.376183 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:01.376209 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:01.376679 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:01.376996 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:01.377784 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.379252 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.380138 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.380871 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.381032 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:01.383791 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:01.383797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:01.383814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:01.383849 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:01.385998 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:01.386536 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:01.386544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:01.386552 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:01.386695 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:01.387036 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:01.387039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:01.387040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:01.387563 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:01.387786 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:01.388063 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 08:24:01.388565 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
W20260808 08:24:01.392827 5858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:01.392915 5858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:01.392987 5851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:01.393925 5851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:01.399678 5851 catalog_manager.cc:1347] Generated new cluster ID: 0297b8f4f0f547f1884a19e4a6c56e1b
I20260808 08:24:01.399760 5851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:01.409250 5851 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:01.410305 5851 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:01.419304 5851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: Generated new TSK 0
I20260808 08:24:01.420079 5851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:24:01.462339 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36659
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:01.585500 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:01.585848 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:01.585975 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:01.590830 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:01.590922 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:01.591017 5868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:24:01.596702 5868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:01.597923 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:01.599033 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:01.609592 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:01.609650 5868 server_base.cc:1034] running on GCE node
W20260808 08:24:01.609666 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:01.609592 5880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:01.610368 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:01.611061 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:01.612267 5868 hybrid_clock.cc:648] HybridClock initialized: now 1786177441612257 us; error 31 us; skew 500 ppm
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:01.615413 5868 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:24:01.616382 5868 webserver.cc:460] Webserver started at http://127.12.13.1:45655/ using document root <none> and password file <none>
I20260808 08:24:01.616998 5868 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:01.617093 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:01.617352 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:01.620059 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "0e3586fc2d4642f68e829ed915d56d39"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "c411f19104ea472225448e784c36978c"
server_key_iv: "3892d474488f0cd0162bff4d88fa9711"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.620739 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "0e3586fc2d4642f68e829ed915d56d39"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "c411f19104ea472225448e784c36978c"
server_key_iv: "3892d474488f0cd0162bff4d88fa9711"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.624958 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:24:01.627936 5892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.629513 5868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:24:01.629685 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "0e3586fc2d4642f68e829ed915d56d39"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "c411f19104ea472225448e784c36978c"
server_key_iv: "3892d474488f0cd0162bff4d88fa9711"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.629823 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:01.644539 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:01.648115 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:01.648355 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:01.649019 5868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:01.649961 5868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:01.650038 5868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.650115 5868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:01.650168 5868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.660095 5868 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:42647
I20260808 08:24:01.660121 6062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:42647 every 8 connection(s)
I20260808 08:24:01.661144 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:24:01.671335 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 5868
I20260808 08:24:01.671482 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:24:01.671875 12340 external_mini_cluster.cc:1428] Setting key ee3bdbbb2ec06d080f6ea452661cbda6
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:01.677079 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:01.663338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56407 (local address 127.12.13.62:36659)
0808 08:24:01.663509 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:01.663515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:01.664584 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 08:24:01.666115 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:01.666118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:01.666174 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:01.666260 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:01.667689 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.668350 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.669573 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.669822 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.670001 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:01.673045 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:01.673048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:01.673051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:01.673077 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:01.674752 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:01.675379 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:01.675382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:01.675384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:01.675482 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:01.675861 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:01.675866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:01.675868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:01.676067 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:01.676184 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:01.676757 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 08:24:01.676900 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 08:24:01.678131 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:01.663804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36659 (local address 127.12.13.1:56407)
0808 08:24:01.664406 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:01.664454 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:01.665820 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:01.666484 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:01.666491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:01.666896 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:01.667494 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:01.667509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.668519 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.668522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:01.669437 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:01.669447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.670063 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.670066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:01.670285 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:01.671013 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:01.671048 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:01.672878 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:01.674884 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:01.674890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:01.674908 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:01.675234 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:01.675583 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:01.675586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:01.675587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:01.675758 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:01.676189 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:01.676199 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:01.676487 (+ 288us) client_negotiation.cc:769] Sending connection context
0808 08:24:01.676713 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 08:24:01.677130 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260808 08:24:01.679709 6063 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36659
I20260808 08:24:01.679993 6063 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:01.680549 6063 heartbeater.cc:511] Master 127.12.13.62:36659 requested a full tablet report, sending...
I20260808 08:24:01.682427 5784 ts_manager.cc:194] Registered new tserver with Master: 0e3586fc2d4642f68e829ed915d56d39 (127.12.13.1:42647)
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:24:01.683858 5784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:56407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:24:01.705670 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36659
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:01.828461 6071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:01.828749 6071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:01.828795 6071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:01.832952 6071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:01.833016 6071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:01.833077 6071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:24:01.838632 6071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:01.839983 6071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:01.841315 6071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:01.851928 6076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:01.851982 6077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:01.851971 6071 server_base.cc:1034] running on GCE node
W20260808 08:24:01.851928 6080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:01.852783 6071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:01.853525 6071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:01.854718 6071 hybrid_clock.cc:648] HybridClock initialized: now 1786177441854704 us; error 27 us; skew 500 ppm
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:01.857723 6071 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:24:01.858860 6071 webserver.cc:460] Webserver started at http://127.12.13.2:45021/ using document root <none> and password file <none>
I20260808 08:24:01.859548 6071 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:01.859650 6071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:01.859905 6071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:01.862202 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "5c16de5be6cb445987b611fb3570e3ca"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "d11a6ce92a7f957a1d1743aefdf4aac3"
server_key_iv: "5881566566a78fc198cffb5fa30f1235"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.862819 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "5c16de5be6cb445987b611fb3570e3ca"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "d11a6ce92a7f957a1d1743aefdf4aac3"
server_key_iv: "5881566566a78fc198cffb5fa30f1235"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.867358 6071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:24:01.870272 6087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.871670 6071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:24:01.871822 6071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "5c16de5be6cb445987b611fb3570e3ca"
format_stamp: "Formatted at 2026-08-08 08:24:01 on dist-test-slave-6ntk"
server_key: "d11a6ce92a7f957a1d1743aefdf4aac3"
server_key_iv: "5881566566a78fc198cffb5fa30f1235"
server_key_version: "encryptionkey@0"
I20260808 08:24:01.871948 6071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:01.885317 6071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:01.888774 6071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:01.889086 6071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:01.889721 6071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:01.890738 6071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:01.890787 6071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.890830 6071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:01.890846 6071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:01.900462 6071 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:40539
I20260808 08:24:01.900501 6257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:40539 every 8 connection(s)
I20260808 08:24:01.901772 6071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:24:01.904318 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 6071
I20260808 08:24:01.904481 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:24:01.904790 12340 external_mini_cluster.cc:1428] Setting key fb3046c30055bf50373d6984d7de80e9
Aug 08 08:24:01 dist-test-slave-6ntk krb5kdc[5728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
I20260808 08:24:01.918409 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:01.904131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:52423 (local address 127.12.13.62:36659)
0808 08:24:01.904336 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:01.904341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:01.905294 (+ 953us) server_negotiation.cc:407] Connection header received
0808 08:24:01.906942 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:01.906947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:01.907014 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:01.907111 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:01.908356 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.909096 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.910962 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.911190 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.911274 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:01.914513 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:01.914516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:01.914518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:01.914544 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:01.916072 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:01.916700 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:01.916703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:01.916704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:01.916783 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:01.917168 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:01.917172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:01.917173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:01.917377 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:01.917489 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:01.918101 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 08:24:01.918233 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
I20260808 08:24:01.919400 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:01.904588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36659 (local address 127.12.13.2:52423)
0808 08:24:01.905120 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:01.905175 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:01.906713 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:01.907273 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:01.907281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:01.907674 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:01.908202 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:01.908216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.909248 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.909255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:01.910833 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:01.910850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:01.911361 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:01.911369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:01.911617 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:01.912470 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:01.912498 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:01.914306 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:01.916207 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:01.916213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:01.916233 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:01.916571 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:01.916894 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:01.916897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:01.916899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:01.917058 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:01.917490 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:01.917499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:01.917786 (+ 287us) client_negotiation.cc:769] Sending connection context
0808 08:24:01.918038 (+ 252us) client_negotiation.cc:240] Negotiation successful
0808 08:24:01.918416 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":142,"threads_started":1}
I20260808 08:24:01.920706 6259 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36659
I20260808 08:24:01.921006 6259 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:01.921567 6259 heartbeater.cc:511] Master 127.12.13.62:36659 requested a full tablet report, sending...
I20260808 08:24:01.922739 5784 ts_manager.cc:194] Registered new tserver with Master: 5c16de5be6cb445987b611fb3570e3ca (127.12.13.2:40539)
I20260808 08:24:01.923332 5784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:52423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:24:01.938356 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36659
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:02.061549 6268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:02.061854 6268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:02.062001 6268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:02.066282 6268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:02.066349 6268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:02.066411 6268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:24:02.072752 6268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:02.074018 6268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:02.075217 6268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:02.085825 6278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.085825 6277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.085942 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:02.086431 6268 server_base.cc:1034] running on GCE node
I20260808 08:24:02.086926 6268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:02.087767 6268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:02.088953 6268 hybrid_clock.cc:648] HybridClock initialized: now 1786177442088929 us; error 37 us; skew 500 ppm
Aug 08 08:24:02 dist-test-slave-6ntk krb5kdc[5728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:02.092391 6268 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:24:02.093457 6268 webserver.cc:460] Webserver started at http://127.12.13.3:41989/ using document root <none> and password file <none>
I20260808 08:24:02.094233 6268 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:02.094303 6268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:02.094486 6268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:02.096858 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "5105490718a243b89d85933de8371318"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "5e8a4ec8e401d95b21cee4c6ce647bd8"
server_key_iv: "9cb0afdcfd59a71c5e35b732174672c2"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.097398 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "5105490718a243b89d85933de8371318"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "5e8a4ec8e401d95b21cee4c6ce647bd8"
server_key_iv: "9cb0afdcfd59a71c5e35b732174672c2"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.101555 6268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260808 08:24:02.104568 6289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.106225 6268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:02.106372 6268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "5105490718a243b89d85933de8371318"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "5e8a4ec8e401d95b21cee4c6ce647bd8"
server_key_iv: "9cb0afdcfd59a71c5e35b732174672c2"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.106518 6268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:02.122047 6268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:02.125710 6268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:02.126022 6268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:02.126799 6268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:02.127862 6268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:02.127938 6268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.128013 6268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:02.128065 6268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.138645 6268 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:36999
I20260808 08:24:02.138669 6448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:36999 every 8 connection(s)
I20260808 08:24:02.139652 6268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:24:02.146993 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 6268
I20260808 08:24:02.147113 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:24:02.147609 12340 external_mini_cluster.cc:1428] Setting key 74a064e2ce2bf3710be4ceece44e51f2
Aug 08 08:24:02 dist-test-slave-6ntk krb5kdc[5728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:02.155120 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.141708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:46887 (local address 127.12.13.62:36659)
0808 08:24:02.141923 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.141928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.142895 (+ 967us) server_negotiation.cc:407] Connection header received
0808 08:24:02.144358 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.144362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.144426 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.144541 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:02.145802 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.146412 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.147789 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.148080 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.148203 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.151281 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:02.151285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:02.151287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:02.151315 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:02.152997 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:02.153524 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:02.153527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:02.153528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:02.153610 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:02.154007 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:02.154010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:02.154011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:02.154211 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:02.154318 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.154884 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.154992 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 08:24:02.155938 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.142173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36659 (local address 127.12.13.3:46887)
0808 08:24:02.142670 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.142742 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.144107 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.144738 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.144745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.145124 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:02.145628 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.145643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.146552 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.146562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.147624 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.147636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.148256 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.148261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.148467 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.149366 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:02.149383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:02.151046 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:02.153105 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:02.153110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:02.153124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:02.153432 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:02.153695 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:02.153697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:02.153698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:02.153845 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:02.154357 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:02.154363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:02.154619 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.154779 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.155047 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260808 08:24:02.157207 6449 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36659
I20260808 08:24:02.157531 6449 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:02.158181 6449 heartbeater.cc:511] Master 127.12.13.62:36659 requested a full tablet report, sending...
I20260808 08:24:02.159422 5784 ts_manager.cc:194] Registered new tserver with Master: 5105490718a243b89d85933de8371318 (127.12.13.3:36999)
I20260808 08:24:02.159999 5784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:46887
I20260808 08:24:02.162657 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:24:02.172959 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.164753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.12.13.62:36659)
0808 08:24:02.164914 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.164918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.165118 (+ 200us) server_negotiation.cc:407] Connection header received
0808 08:24:02.165279 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.165282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.165327 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.165417 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:02.166260 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.166794 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.167769 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.167987 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.168080 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.169517 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:02.169520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:02.169522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:02.169545 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:02.171257 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:02.171743 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:02.171746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:02.171747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:02.171825 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:02.172061 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:02.172063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:02.172064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:02.172343 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:02.172454 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.172728 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.172828 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 08:24:02.176098 5784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:24:02.179459 5784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:24:02.196723 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.189555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40539 (local address 127.0.0.1:50488)
0808 08:24:02.190521 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.190533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.190613 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.191494 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.191500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.191527 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:02.191689 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.191698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.193909 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.193912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.194934 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.194944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.196093 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.196097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.196260 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.196311 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.196416 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.196538 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260808 08:24:02.197165 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.189516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:42647 (local address 127.0.0.1:52720)
0808 08:24:02.190028 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.190099 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.190290 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.192020 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.192023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.192043 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:02.192140 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.192146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.194278 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.194282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.195589 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.195601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.196719 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.196723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.196813 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.196840 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.196930 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.197001 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":46,"thread_start_us":110,"threads_started":1}
I20260808 08:24:02.197211 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.191205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.12.13.1:42647)
0808 08:24:02.191610 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.191617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.191638 (+ 21us) server_negotiation.cc:407] Connection header received
0808 08:24:02.191709 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.191713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.191858 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.192035 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:02.193073 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.194141 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.195742 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.196584 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.196689 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.196854 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.196961 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.197059 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":97,"threads_started":1}
I20260808 08:24:02.197693 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.190480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.12.13.2:40539)
0808 08:24:02.190916 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.190930 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.190955 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:24:02.191037 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.191042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.191272 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.192294 (+ 1022us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:02.192477 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.193718 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.195152 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.195955 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.197174 (+ 1219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.197375 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.197433 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.197535 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":220,"threads_started":1}
I20260808 08:24:02.197765 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.191090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36999 (local address 127.0.0.1:60332)
0808 08:24:02.192261 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.192273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.192362 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.193562 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.193565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.193586 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:02.193683 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.193690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.195128 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.195131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.196200 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.196214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.197361 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.197365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.197458 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.197487 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.197563 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.197641 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260808 08:24:02.198173 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.192712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.12.13.3:36999)
0808 08:24:02.193115 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.193127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.193150 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:24:02.193226 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.193230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.193404 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.193556 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:02.193815 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.194964 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.196311 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.197038 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.197133 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.197294 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.197953 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.198049 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":148,"threads_started":1}
I20260808 08:24:02.200675 6360 tablet_service.cc:1467] Processing CreateTablet for tablet 9138866eee1a4631b93c646e688f4afc (DEFAULT_TABLE table=test-table [id=1c537e11be52492081ade2981e502eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:02.200701 5965 tablet_service.cc:1467] Processing CreateTablet for tablet 9138866eee1a4631b93c646e688f4afc (DEFAULT_TABLE table=test-table [id=1c537e11be52492081ade2981e502eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:02.200701 6151 tablet_service.cc:1467] Processing CreateTablet for tablet 9138866eee1a4631b93c646e688f4afc (DEFAULT_TABLE table=test-table [id=1c537e11be52492081ade2981e502eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:02.201815 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9138866eee1a4631b93c646e688f4afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:02.201822 6151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9138866eee1a4631b93c646e688f4afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:02.201833 6360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9138866eee1a4631b93c646e688f4afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:02.208700 6468 tablet_bootstrap.cc:492] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Bootstrap starting.
I20260808 08:24:02.211606 6468 tablet_bootstrap.cc:654] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:02.212900 6468 log.cc:824] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:02.213631 6469 tablet_bootstrap.cc:492] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39: Bootstrap starting.
I20260808 08:24:02.214268 6470 tablet_bootstrap.cc:492] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Bootstrap starting.
I20260808 08:24:02.215332 6468 tablet_bootstrap.cc:492] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: No bootstrap required, opened a new log
I20260808 08:24:02.215550 6468 ts_tablet_manager.cc:1397] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 08:24:02.215826 6469 tablet_bootstrap.cc:654] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:02.216596 6469 log.cc:824] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:02.216900 6470 tablet_bootstrap.cc:654] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:02.218390 6470 log.cc:824] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:02.220593 6470 tablet_bootstrap.cc:492] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: No bootstrap required, opened a new log
I20260808 08:24:02.220631 6469 tablet_bootstrap.cc:492] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39: No bootstrap required, opened a new log
I20260808 08:24:02.220831 6470 ts_tablet_manager.cc:1397] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 08:24:02.220835 6469 ts_tablet_manager.cc:1397] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260808 08:24:02.220708 6468 raft_consensus.cc:348] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.221045 6468 raft_consensus.cc:374] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:02.221174 6468 raft_consensus.cc:729] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5105490718a243b89d85933de8371318, State: Initialized, Role: FOLLOWER
I20260808 08:24:02.221721 6468 consensus_queue.cc:260] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.222534 6449 heartbeater.cc:503] Master 127.12.13.62:36659 was elected leader, sending a full tablet report...
I20260808 08:24:02.222966 6468 ts_tablet_manager.cc:1428] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260808 08:24:02.224781 6470 raft_consensus.cc:348] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.225047 6470 raft_consensus.cc:374] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:02.225142 6470 raft_consensus.cc:729] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c16de5be6cb445987b611fb3570e3ca, State: Initialized, Role: FOLLOWER
I20260808 08:24:02.225625 6470 consensus_queue.cc:260] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.225699 6469 raft_consensus.cc:348] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.226163 6469 raft_consensus.cc:374] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:02.226311 6469 raft_consensus.cc:729] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3586fc2d4642f68e829ed915d56d39, State: Initialized, Role: FOLLOWER
I20260808 08:24:02.226744 6470 ts_tablet_manager.cc:1428] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 08:24:02.226840 6469 consensus_queue.cc:260] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.227145 6259 heartbeater.cc:503] Master 127.12.13.62:36659 was elected leader, sending a full tablet report...
I20260808 08:24:02.228072 6469 ts_tablet_manager.cc:1428] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260808 08:24:02.228506 6063 heartbeater.cc:503] Master 127.12.13.62:36659 was elected leader, sending a full tablet report...
I20260808 08:24:02.240365 6480 raft_consensus.cc:482] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:24:02.240602 6480 raft_consensus.cc:504] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.241896 6480 leader_election.cc:288] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c16de5be6cb445987b611fb3570e3ca (127.12.13.2:40539), 5105490718a243b89d85933de8371318 (127.12.13.3:36999)
I20260808 08:24:02.247964 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.242334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:39767 (local address 127.12.13.3:36999)
0808 08:24:02.242484 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.242490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.242812 (+ 322us) server_negotiation.cc:407] Connection header received
0808 08:24:02.242973 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.242976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.243039 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.243136 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:02.243683 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.244545 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.246471 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.247131 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.247272 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.247352 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.247729 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.247817 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 08:24:02.247964 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.242421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:56695 (local address 127.12.13.2:40539)
0808 08:24:02.242547 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.242551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.242931 (+ 380us) server_negotiation.cc:407] Connection header received
0808 08:24:02.242996 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.242999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.243047 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.243136 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:02.243683 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.244486 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.246459 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.247138 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.247272 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.247352 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.247729 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.247817 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:24:02.248003 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.242260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:40539 (local address 127.12.13.1:56695)
0808 08:24:02.242802 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.242815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.242905 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.243193 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.243197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.243245 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:02.243523 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.243530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.244641 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.244645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.246280 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.246290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.247302 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.247306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.247516 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.247572 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.247661 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.247774 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":323,"spinlock_wait_cycles":10752,"thread_start_us":106,"threads_started":1}
I20260808 08:24:02.248003 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.242199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:36999 (local address 127.12.13.1:39767)
0808 08:24:02.242663 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.242681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.242828 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.243193 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.243197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.243244 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:02.243521 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.243530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.244646 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.244648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.246275 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.246294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.247319 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.247323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.247499 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.247578 (+ 79us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.247660 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.247774 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":245,"spinlock_wait_cycles":32896,"thread_start_us":167,"threads_started":1}
I20260808 08:24:02.249011 6182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9138866eee1a4631b93c646e688f4afc" candidate_uuid: "0e3586fc2d4642f68e829ed915d56d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c16de5be6cb445987b611fb3570e3ca" is_pre_election: true
I20260808 08:24:02.249337 6384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9138866eee1a4631b93c646e688f4afc" candidate_uuid: "0e3586fc2d4642f68e829ed915d56d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5105490718a243b89d85933de8371318" is_pre_election: true
I20260808 08:24:02.249477 6182 raft_consensus.cc:2454] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3586fc2d4642f68e829ed915d56d39 in term 0.
I20260808 08:24:02.249675 6384 raft_consensus.cc:2454] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3586fc2d4642f68e829ed915d56d39 in term 0.
I20260808 08:24:02.250061 5896 leader_election.cc:302] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3586fc2d4642f68e829ed915d56d39, 5c16de5be6cb445987b611fb3570e3ca; no voters:
I20260808 08:24:02.250382 6480 raft_consensus.cc:2781] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:24:02.250478 6480 raft_consensus.cc:482] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:24:02.250535 6480 raft_consensus.cc:3037] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:02.251724 6480 raft_consensus.cc:504] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.252512 6480 leader_election.cc:288] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [CANDIDATE]: Term 1 election: Requested vote from peers 5c16de5be6cb445987b611fb3570e3ca (127.12.13.2:40539), 5105490718a243b89d85933de8371318 (127.12.13.3:36999)
I20260808 08:24:02.252612 6384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9138866eee1a4631b93c646e688f4afc" candidate_uuid: "0e3586fc2d4642f68e829ed915d56d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5105490718a243b89d85933de8371318"
I20260808 08:24:02.252775 6384 raft_consensus.cc:3037] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:02.253691 6182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9138866eee1a4631b93c646e688f4afc" candidate_uuid: "0e3586fc2d4642f68e829ed915d56d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c16de5be6cb445987b611fb3570e3ca"
I20260808 08:24:02.253813 6182 raft_consensus.cc:3037] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:02.254305 6384 raft_consensus.cc:2454] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3586fc2d4642f68e829ed915d56d39 in term 1.
I20260808 08:24:02.254724 5897 leader_election.cc:302] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3586fc2d4642f68e829ed915d56d39, 5105490718a243b89d85933de8371318; no voters:
I20260808 08:24:02.254936 6480 raft_consensus.cc:2781] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:02.255122 6182 raft_consensus.cc:2454] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3586fc2d4642f68e829ed915d56d39 in term 1.
I20260808 08:24:02.255254 6480 raft_consensus.cc:686] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [term 1 LEADER]: Becoming Leader. State: Replica: 0e3586fc2d4642f68e829ed915d56d39, State: Running, Role: LEADER
I20260808 08:24:02.255682 6480 consensus_queue.cc:237] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } }
I20260808 08:24:02.259521 5785 catalog_manager.cc:5495] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e3586fc2d4642f68e829ed915d56d39 (127.12.13.1). New cstate: current_term: 1 leader_uuid: "0e3586fc2d4642f68e829ed915d56d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3586fc2d4642f68e829ed915d56d39" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 42647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 } health_report { overall_health: UNKNOWN } } }
I20260808 08:24:02.278024 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.274259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.12.13.1:42647)
0808 08:24:02.274404 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.274408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.274423 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:24:02.274504 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.274506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.274564 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.274660 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:02.274885 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.275420 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.276353 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.276580 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.276731 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.277053 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:02.277505 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:02.277604 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.277772 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.277845 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:24:02.284808 6384 raft_consensus.cc:1261] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 1 FOLLOWER]: Refusing update from remote peer 0e3586fc2d4642f68e829ed915d56d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:02.284803 6182 raft_consensus.cc:1261] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 1 FOLLOWER]: Refusing update from remote peer 0e3586fc2d4642f68e829ed915d56d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:02.285561 6485 consensus_queue.cc:1035] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5105490718a243b89d85933de8371318" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 36999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:02.285801 6480 consensus_queue.cc:1035] T 9138866eee1a4631b93c646e688f4afc P 0e3586fc2d4642f68e829ed915d56d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c16de5be6cb445987b611fb3570e3ca" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:02.293195 6490 mvcc.cc:205] Tried to move back new op lower bound from 7316182803589799936 to 7316182803482361856. Current Snapshot: MvccSnapshot[applied={T|T < 7316182803589799936}]
I20260808 08:24:02.294087 6491 mvcc.cc:205] Tried to move back new op lower bound from 7316182803589799936 to 7316182803482361856. Current Snapshot: MvccSnapshot[applied={T|T < 7316182803589799936}]
I20260808 08:24:02.297472 6494 mvcc.cc:205] Tried to move back new op lower bound from 7316182803589799936 to 7316182803482361856. Current Snapshot: MvccSnapshot[applied={T|T < 7316182803589799936}]
I20260808 08:24:02.302755 5785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:02.302954 5785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:02.305540 5785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7120e60e3f7844319c7be9568ce8833a: Sending DeleteTablet for 3 replicas of tablet 9138866eee1a4631b93c646e688f4afc
I20260808 08:24:02.306509 5965 tablet_service.cc:1514] Processing DeleteTablet for tablet 9138866eee1a4631b93c646e688f4afc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:52720
I20260808 08:24:02.306571 6151 tablet_service.cc:1514] Processing DeleteTablet for tablet 9138866eee1a4631b93c646e688f4afc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50488
I20260808 08:24:02.307123 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 5868
I20260808 08:24:02.307968 6360 tablet_service.cc:1514] Processing DeleteTablet for tablet 9138866eee1a4631b93c646e688f4afc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:02 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:60332
I20260808 08:24:02.310561 6504 tablet_replica.cc:330] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: stopping tablet replica
I20260808 08:24:02.310946 6504 raft_consensus.cc:2229] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:02.311280 6504 raft_consensus.cc:2258] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:02.313529 6504 ts_tablet_manager.cc:1905] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:02.314653 6502 tablet_replica.cc:330] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: stopping tablet replica
I20260808 08:24:02.315011 6502 raft_consensus.cc:2229] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:02.315634 6502 raft_consensus.cc:2258] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:02.317219 6502 ts_tablet_manager.cc:1905] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:02.317795 6504 ts_tablet_manager.cc:1918] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:02.317948 6504 log.cc:1196] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/9138866eee1a4631b93c646e688f4afc
I20260808 08:24:02.318377 6504 ts_tablet_manager.cc:1939] T 9138866eee1a4631b93c646e688f4afc P 5105490718a243b89d85933de8371318: Deleting consensus metadata
I20260808 08:24:02.319835 6502 ts_tablet_manager.cc:1918] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:02.320051 6502 log.cc:1196] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/9138866eee1a4631b93c646e688f4afc
I20260808 08:24:02.320019 5765 catalog_manager.cc:4841] TS 5105490718a243b89d85933de8371318 (127.12.13.3:36999): tablet 9138866eee1a4631b93c646e688f4afc (table test-table [id=1c537e11be52492081ade2981e502eb1]) successfully deleted
I20260808 08:24:02.320441 6502 ts_tablet_manager.cc:1939] T 9138866eee1a4631b93c646e688f4afc P 5c16de5be6cb445987b611fb3570e3ca: Deleting consensus metadata
I20260808 08:24:02.321570 5764 catalog_manager.cc:4841] TS 5c16de5be6cb445987b611fb3570e3ca (127.12.13.2:40539): tablet 9138866eee1a4631b93c646e688f4afc (table test-table [id=1c537e11be52492081ade2981e502eb1]) successfully deleted
I20260808 08:24:02.323163 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 6071
W20260808 08:24:02.323159 5765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:42647 (error 108)
I20260808 08:24:02.332938 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.332419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:42647 (local address 127.0.0.1:52746)
0808 08:24:02.332698 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.332810 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:42647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260808 08:24:02.333186 5765 catalog_manager.cc:4584] TS 0e3586fc2d4642f68e829ed915d56d39 (127.12.13.1:42647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9138866eee1a4631b93c646e688f4afc: Network error: Client connection negotiation failed: client connection to 127.12.13.1:42647: connect: Connection refused (error 111)
I20260808 08:24:02.339546 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 6268
I20260808 08:24:02.355949 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 5742
2026-08-08T08:24:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2637 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T08:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:02Z Disabled control of system clock
I20260808 08:24:02.399654 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:45811
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:44901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:45811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 08:24:02.517444 6510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:02.517743 6510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:02.517781 6510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:02.521921 6510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:02.521999 6510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:02.522015 6510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:02.522032 6510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:02.522048 6510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:02.527185 6510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:45811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:02.528442 6510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:02.529974 6510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:02.538775 6517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.538789 6518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.538925 6523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:02.539301 6510 server_base.cc:1034] running on GCE node
I20260808 08:24:02.539938 6510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:02.541150 6510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:02.542343 6510 hybrid_clock.cc:648] HybridClock initialized: now 1786177442542317 us; error 52 us; skew 500 ppm
I20260808 08:24:02.544293 6510 webserver.cc:460] Webserver started at http://127.12.13.62:38973/ using document root <none> and password file <none>
I20260808 08:24:02.544986 6510 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:02.545084 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:02.545346 6510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:02.547655 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "165d595ee4d34a24b49192a7c1ae5178"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "1103b1f7b9648735044a9b3d0e0e4d1c"
server_key_iv: "2734b58774132009321d20434e5e3ef6"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.548264 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "165d595ee4d34a24b49192a7c1ae5178"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "1103b1f7b9648735044a9b3d0e0e4d1c"
server_key_iv: "2734b58774132009321d20434e5e3ef6"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.552304 6510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 08:24:02.555051 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.556329 6510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:24:02.556515 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "165d595ee4d34a24b49192a7c1ae5178"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "1103b1f7b9648735044a9b3d0e0e4d1c"
server_key_iv: "2734b58774132009321d20434e5e3ef6"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.556644 6510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:02.574090 6510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:02.574927 6510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:02.575146 6510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:02.582823 6510 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:45811
I20260808 08:24:02.582830 6602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:45811 every 8 connection(s)
I20260808 08:24:02.583833 6510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:02.586557 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 6510
I20260808 08:24:02.586707 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:02.587013 12340 external_mini_cluster.cc:1428] Setting key 3b299bdd934ead1f2e60b11724246736
I20260808 08:24:02.587324 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:02.595050 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178: Bootstrap starting.
I20260808 08:24:02.597554 6604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:02.597796 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.588554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.12.13.62:45811)
0808 08:24:02.589265 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.589286 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.589331 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:24:02.590291 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.590316 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.590841 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.591229 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:02.591700 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.593178 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.594150 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.594835 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.594998 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.595963 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:02.595968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:02.595981 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:02.596396 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:02.596454 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:02.596476 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:02.596593 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.596713 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.597002 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":110,"threads_started":1}
I20260808 08:24:02.598452 6604 log.cc:824] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:02.600559 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178: No bootstrap required, opened a new log
I20260808 08:24:02.604606 6604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d595ee4d34a24b49192a7c1ae5178" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 45811 } }
I20260808 08:24:02.604877 6604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:02.604976 6604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165d595ee4d34a24b49192a7c1ae5178, State: Initialized, Role: FOLLOWER
I20260808 08:24:02.605520 6604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d595ee4d34a24b49192a7c1ae5178" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 45811 } }
I20260808 08:24:02.605700 6604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:02.605775 6604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:02.605927 6604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:02.606974 6604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d595ee4d34a24b49192a7c1ae5178" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 45811 } }
I20260808 08:24:02.607354 6604 leader_election.cc:302] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165d595ee4d34a24b49192a7c1ae5178; no voters:
I20260808 08:24:02.607692 6604 leader_election.cc:288] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:02.607882 6612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:02.608177 6612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [term 1 LEADER]: Becoming Leader. State: Replica: 165d595ee4d34a24b49192a7c1ae5178, State: Running, Role: LEADER
I20260808 08:24:02.608533 6612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d595ee4d34a24b49192a7c1ae5178" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 45811 } }
I20260808 08:24:02.609088 6604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:02.610427 6614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165d595ee4d34a24b49192a7c1ae5178. Latest consensus state: current_term: 1 leader_uuid: "165d595ee4d34a24b49192a7c1ae5178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d595ee4d34a24b49192a7c1ae5178" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 45811 } } }
I20260808 08:24:02.610466 6613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165d595ee4d34a24b49192a7c1ae5178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d595ee4d34a24b49192a7c1ae5178" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 45811 } } }
I20260808 08:24:02.610565 6614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:02.610574 6613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:02.610986 6624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:02.614636 6624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:02.621153 6624 catalog_manager.cc:1347] Generated new cluster ID: 440c657eb43c4f48bbd2055fb77cfd30
I20260808 08:24:02.621249 6624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:02.637476 6624 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:02.638634 6624 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:02.643778 6624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178: Generated new TSK 0
I20260808 08:24:02.644575 6624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:02.649319 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:45811
--builtin_ntp_servers=127.12.13.20:44901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 08:24:02.777668 6639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:02.778075 6639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:02.778126 6639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:02.782266 6639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:02.782327 6639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:02.782383 6639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:24:02.787283 6639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:02.788468 6639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:02.789992 6639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:02.800477 6647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.800637 6649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.800597 6645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:02.800809 6639 server_base.cc:1034] running on GCE node
I20260808 08:24:02.801260 6639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:02.801991 6639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:02.803158 6639 hybrid_clock.cc:648] HybridClock initialized: now 1786177442803121 us; error 49 us; skew 500 ppm
I20260808 08:24:02.805063 6639 webserver.cc:460] Webserver started at http://127.12.13.1:39337/ using document root <none> and password file <none>
I20260808 08:24:02.805672 6639 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:02.805737 6639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:02.806046 6639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:02.808240 6639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "d3aa8d9b82e54e35882e1b3c224f7bab"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "9d36bc761cd418af729711cf62e78f97"
server_key_iv: "a79bb644400d8410b523a4977728b7e9"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.808883 6639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "d3aa8d9b82e54e35882e1b3c224f7bab"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "9d36bc761cd418af729711cf62e78f97"
server_key_iv: "a79bb644400d8410b523a4977728b7e9"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.813149 6639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:24:02.815809 6656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.817107 6639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:24:02.817256 6639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "d3aa8d9b82e54e35882e1b3c224f7bab"
format_stamp: "Formatted at 2026-08-08 08:24:02 on dist-test-slave-6ntk"
server_key: "9d36bc761cd418af729711cf62e78f97"
server_key_iv: "a79bb644400d8410b523a4977728b7e9"
server_key_version: "encryptionkey@0"
I20260808 08:24:02.817401 6639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:02.829054 6639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:02.829834 6639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:02.830127 6639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:02.830790 6639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:02.831874 6639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:02.831948 6639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.832024 6639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:02.832075 6639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:02.842224 6639 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:33489
I20260808 08:24:02.842249 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:33489 every 8 connection(s)
I20260808 08:24:02.843236 6639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:24:02.848486 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 6639
I20260808 08:24:02.848588 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:24:02.848902 12340 external_mini_cluster.cc:1428] Setting key b71c965c36fe328558bd3be548cda5bd
I20260808 08:24:02.851084 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:45811
--builtin_ntp_servers=127.12.13.20:44901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 08:24:02.853950 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.845307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:35589 (local address 127.12.13.62:45811)
0808 08:24:02.845473 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:02.845477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:02.846667 (+ 1190us) server_negotiation.cc:407] Connection header received
0808 08:24:02.848149 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:02.848153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:02.848217 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:02.848345 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:02.849440 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.850161 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.851574 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.851990 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.852118 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.853401 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:02.853404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:02.853406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:02.853418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:02.853438 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:02.853446 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:02.853525 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:02.853699 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 08:24:02.853770 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 08:24:02.854929 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:02.845733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:45811 (local address 127.12.13.1:35589)
0808 08:24:02.846492 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:02.846537 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:02.847907 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:02.848568 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:02.848579 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:02.848696 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:02.849282 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.849297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.850397 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.850400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.851445 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:02.851456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:02.852151 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:02.852153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:02.852345 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:02.853084 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:02.853085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:02.853130 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:02.853223 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:02.853529 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:02.853542 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:02.853548 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:24:02.853685 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 08:24:02.854105 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":163,"threads_started":1}
I20260808 08:24:02.856362 6827 heartbeater.cc:347] Connected to a master server at 127.12.13.62:45811
I20260808 08:24:02.856629 6827 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:02.857231 6827 heartbeater.cc:511] Master 127.12.13.62:45811 requested a full tablet report, sending...
I20260808 08:24:02.859213 6556 ts_manager.cc:194] Registered new tserver with Master: d3aa8d9b82e54e35882e1b3c224f7bab (127.12.13.1:33489)
I20260808 08:24:02.861119 6556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.1:35589
W20260808 08:24:02.973711 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:02.974138 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:02.974189 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:02.978294 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:02.978356 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:02.978413 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:24:02.983471 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:02.984684 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:02.986207 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:02.996825 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.996855 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:02.996941 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:02.997076 6832 server_base.cc:1034] running on GCE node
I20260808 08:24:02.997529 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:02.998266 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:02.999444 6832 hybrid_clock.cc:648] HybridClock initialized: now 1786177442999425 us; error 29 us; skew 500 ppm
I20260808 08:24:03.001492 6832 webserver.cc:460] Webserver started at http://127.12.13.2:36285/ using document root <none> and password file <none>
I20260808 08:24:03.002179 6832 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:03.002246 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:03.002516 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:03.004678 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "1d464b9d960e4a5e85e15fca59cc986a"
format_stamp: "Formatted at 2026-08-08 08:24:03 on dist-test-slave-6ntk"
server_key: "a58ff5717b008a11bb1f2c08c9cc3ca9"
server_key_iv: "24021d958e55db352e966a2920e99405"
server_key_version: "encryptionkey@0"
I20260808 08:24:03.005308 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "1d464b9d960e4a5e85e15fca59cc986a"
format_stamp: "Formatted at 2026-08-08 08:24:03 on dist-test-slave-6ntk"
server_key: "a58ff5717b008a11bb1f2c08c9cc3ca9"
server_key_iv: "24021d958e55db352e966a2920e99405"
server_key_version: "encryptionkey@0"
I20260808 08:24:03.009620 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:03.012576 6856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:03.014065 6832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:03.014200 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "1d464b9d960e4a5e85e15fca59cc986a"
format_stamp: "Formatted at 2026-08-08 08:24:03 on dist-test-slave-6ntk"
server_key: "a58ff5717b008a11bb1f2c08c9cc3ca9"
server_key_iv: "24021d958e55db352e966a2920e99405"
server_key_version: "encryptionkey@0"
I20260808 08:24:03.014329 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:03.032382 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:03.033200 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:03.033432 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:03.034201 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:03.035233 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:03.035281 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:03.035326 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:03.035382 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:03.045480 6832 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:43267
I20260808 08:24:03.045506 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:43267 every 8 connection(s)
I20260808 08:24:03.046523 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:24:03.050185 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 6832
I20260808 08:24:03.050333 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:24:03.050668 12340 external_mini_cluster.cc:1428] Setting key 8fa5df5b512aa03b91350622e3e61683
I20260808 08:24:03.053130 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:45811
--builtin_ntp_servers=127.12.13.20:44901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 08:24:03.057665 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.048524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:51573 (local address 127.12.13.62:45811)
0808 08:24:03.048708 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.048713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.049850 (+ 1137us) server_negotiation.cc:407] Connection header received
0808 08:24:03.051848 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.051851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.051899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.051970 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:03.053197 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.053862 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.055581 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.055815 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.055933 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.057116 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:03.057119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:03.057121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:03.057134 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:03.057153 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:03.057161 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:03.057245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.057449 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.057514 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 08:24:03.058652 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.048957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:45811 (local address 127.12.13.2:51573)
0808 08:24:03.049667 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.049720 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.051598 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.052202 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.052211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.052300 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:03.053035 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.053059 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.054029 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.054035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.055440 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.055454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.055979 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.055984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.056155 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.056836 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:03.056838 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:03.056880 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:03.056947 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:03.057247 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:03.057260 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:03.057268 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.057448 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.057755 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260808 08:24:03.059974 7012 heartbeater.cc:347] Connected to a master server at 127.12.13.62:45811
I20260808 08:24:03.060295 7012 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:03.061111 7012 heartbeater.cc:511] Master 127.12.13.62:45811 requested a full tablet report, sending...
I20260808 08:24:03.062882 6556 ts_manager.cc:194] Registered new tserver with Master: 1d464b9d960e4a5e85e15fca59cc986a (127.12.13.2:43267)
I20260808 08:24:03.063829 6556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.2:51573
W20260808 08:24:03.178658 7018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:03.178951 7018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:03.178992 7018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:03.183167 7018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:03.183228 7018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:03.183284 7018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:24:03.188699 7018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:03.190222 7018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:03.192060 7018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:03.203713 7029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:03.203653 7028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:03.204063 7018 server_base.cc:1034] running on GCE node
W20260808 08:24:03.204144 7032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:03.204556 7018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:03.205212 7018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:03.206440 7018 hybrid_clock.cc:648] HybridClock initialized: now 1786177443206418 us; error 33 us; skew 500 ppm
I20260808 08:24:03.208565 7018 webserver.cc:460] Webserver started at http://127.12.13.3:35317/ using document root <none> and password file <none>
I20260808 08:24:03.209223 7018 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:03.209345 7018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:03.209619 7018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:03.212134 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "e7074660fb8b43a19681d85a5975468b"
format_stamp: "Formatted at 2026-08-08 08:24:03 on dist-test-slave-6ntk"
server_key: "7e0b44370aa62b5a274eff9e44590e68"
server_key_iv: "cbf3bdbd8cc26eef89539a88f7c18876"
server_key_version: "encryptionkey@0"
I20260808 08:24:03.212771 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "e7074660fb8b43a19681d85a5975468b"
format_stamp: "Formatted at 2026-08-08 08:24:03 on dist-test-slave-6ntk"
server_key: "7e0b44370aa62b5a274eff9e44590e68"
server_key_iv: "cbf3bdbd8cc26eef89539a88f7c18876"
server_key_version: "encryptionkey@0"
I20260808 08:24:03.217080 7018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:03.219936 7043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:03.221422 7018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 08:24:03.221575 7018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "e7074660fb8b43a19681d85a5975468b"
format_stamp: "Formatted at 2026-08-08 08:24:03 on dist-test-slave-6ntk"
server_key: "7e0b44370aa62b5a274eff9e44590e68"
server_key_iv: "cbf3bdbd8cc26eef89539a88f7c18876"
server_key_version: "encryptionkey@0"
I20260808 08:24:03.221741 7018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:03.245652 7018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:03.246569 7018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:03.246809 7018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:03.247503 7018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:03.248579 7018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:03.248652 7018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:03.248730 7018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:03.248772 7018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:03.259016 7018 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:34461
I20260808 08:24:03.259075 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:34461 every 8 connection(s)
I20260808 08:24:03.260051 7018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:24:03.266765 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 7018
I20260808 08:24:03.266911 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:24:03.267251 12340 external_mini_cluster.cc:1428] Setting key 54216e1d208c01700d64d5b46e732442
I20260808 08:24:03.271776 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.262440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:40783 (local address 127.12.13.62:45811)
0808 08:24:03.262611 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.262614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.263867 (+ 1253us) server_negotiation.cc:407] Connection header received
0808 08:24:03.265461 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.265464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.265521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.265646 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:03.267053 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.267729 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.269380 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.269780 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.269953 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.271297 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:03.271300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:03.271303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:03.271316 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:03.271337 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:03.271346 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:03.271415 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.271573 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.271631 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 08:24:03.272778 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.262836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:45811 (local address 127.12.13.3:40783)
0808 08:24:03.263677 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.263724 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.265153 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.265853 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.265861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.266043 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:03.266862 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.266890 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.267908 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.267912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.269233 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.269244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.270055 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.270061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.270259 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.270976 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:03.270978 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:03.271053 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:03.271127 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:03.271432 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:03.271440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:03.271444 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.271549 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.271826 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":155,"threads_started":1}
I20260808 08:24:03.274163 7202 heartbeater.cc:347] Connected to a master server at 127.12.13.62:45811
I20260808 08:24:03.274448 7202 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:03.275068 7202 heartbeater.cc:511] Master 127.12.13.62:45811 requested a full tablet report, sending...
I20260808 08:24:03.276300 6556 ts_manager.cc:194] Registered new tserver with Master: e7074660fb8b43a19681d85a5975468b (127.12.13.3:34461)
I20260808 08:24:03.276922 6556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.3:40783
I20260808 08:24:03.281972 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:24:03.288065 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.284114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.12.13.62:45811)
0808 08:24:03.284343 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.284348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.284525 (+ 177us) server_negotiation.cc:407] Connection header received
0808 08:24:03.284604 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.284606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.284652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.284729 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:03.285008 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.285589 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.286505 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.286729 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.286823 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.287569 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:03.287571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:03.287574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:03.287585 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:03.287600 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:03.287607 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:03.287663 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.287781 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.287875 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260808 08:24:03.291375 6556 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:24:03.294840 6556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:24:03.311596 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.304421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:33489 (local address 127.0.0.1:51164)
0808 08:24:03.305312 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.305372 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.305510 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.306708 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.306717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.306747 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:03.306963 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.306972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.308829 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.308832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.310112 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.310124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.311254 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.311259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.311365 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.311386 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.311452 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.311507 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":101,"thread_start_us":90,"threads_started":1}
I20260808 08:24:03.311596 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.304619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:34461 (local address 127.0.0.1:36542)
0808 08:24:03.305461 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.305476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.305590 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.306762 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.306765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.306788 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:03.306970 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.306977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.308046 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.308052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.309595 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.309606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.310931 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.310937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.311115 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.311156 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.311280 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.311411 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":127,"spinlock_wait_cycles":10752,"thread_start_us":111,"threads_started":1}
I20260808 08:24:03.311942 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.305205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:43267 (local address 127.0.0.1:58780)
0808 08:24:03.305658 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.305675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.305916 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.307090 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.307093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.307109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:03.307198 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.307203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.308735 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.308743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.310054 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.310066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.311485 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.311489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.311602 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.311632 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.311708 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.311785 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":4,"thread_start_us":119,"threads_started":1}
I20260808 08:24:03.312419 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.304609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.12.13.1:33489)
0808 08:24:03.306007 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.306027 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.306056 (+ 29us) server_negotiation.cc:407] Connection header received
0808 08:24:03.306281 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.306289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.306524 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.306702 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:03.307104 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.308688 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.310255 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.310878 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.310988 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.311188 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.312061 (+ 873us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.312202 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":119,"threads_started":1}
I20260808 08:24:03.312458 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.305785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.12.13.2:43267)
0808 08:24:03.306233 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.306245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.306268 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:24:03.306340 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.306345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.306522 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.306703 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:03.307332 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.308572 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.310215 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.311066 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.311195 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.311411 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.312123 (+ 712us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.312269 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":197,"threads_started":1}
I20260808 08:24:03.312740 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.305236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.12.13.3:34461)
0808 08:24:03.306345 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.306355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.306374 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:24:03.306438 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.306442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.306587 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.306929 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:03.307090 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.307925 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.309745 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.310773 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.310894 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.311140 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.312404 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.312529 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":153,"threads_started":1}
I20260808 08:24:03.315562 6735 tablet_service.cc:1467] Processing CreateTablet for tablet 5967cce9727f4163861d906157e671f1 (DEFAULT_TABLE table=test-table [id=b4e2cfefc35f478c87453982bac81695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:03.315608 6922 tablet_service.cc:1467] Processing CreateTablet for tablet 5967cce9727f4163861d906157e671f1 (DEFAULT_TABLE table=test-table [id=b4e2cfefc35f478c87453982bac81695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:03.315917 7112 tablet_service.cc:1467] Processing CreateTablet for tablet 5967cce9727f4163861d906157e671f1 (DEFAULT_TABLE table=test-table [id=b4e2cfefc35f478c87453982bac81695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:03.316828 6922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5967cce9727f4163861d906157e671f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:03.316854 6735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5967cce9727f4163861d906157e671f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:03.316854 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5967cce9727f4163861d906157e671f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:03.323926 7226 tablet_bootstrap.cc:492] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Bootstrap starting.
I20260808 08:24:03.325420 7228 tablet_bootstrap.cc:492] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Bootstrap starting.
I20260808 08:24:03.326288 7227 tablet_bootstrap.cc:492] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: Bootstrap starting.
I20260808 08:24:03.327181 7226 tablet_bootstrap.cc:654] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:03.327970 7228 tablet_bootstrap.cc:654] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:03.328087 7226 log.cc:824] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:03.328730 7227 tablet_bootstrap.cc:654] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:03.328886 7228 log.cc:824] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:03.329571 7227 log.cc:824] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:03.330451 7226 tablet_bootstrap.cc:492] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: No bootstrap required, opened a new log
I20260808 08:24:03.330686 7226 ts_tablet_manager.cc:1397] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 08:24:03.331087 7228 tablet_bootstrap.cc:492] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: No bootstrap required, opened a new log
I20260808 08:24:03.331307 7228 ts_tablet_manager.cc:1397] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:03.331784 7227 tablet_bootstrap.cc:492] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: No bootstrap required, opened a new log
I20260808 08:24:03.331995 7227 ts_tablet_manager.cc:1397] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:03.334689 7226 raft_consensus.cc:348] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.334971 7226 raft_consensus.cc:374] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:03.335028 7226 raft_consensus.cc:729] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d464b9d960e4a5e85e15fca59cc986a, State: Initialized, Role: FOLLOWER
I20260808 08:24:03.335520 7226 consensus_queue.cc:260] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.336247 7012 heartbeater.cc:503] Master 127.12.13.62:45811 was elected leader, sending a full tablet report...
I20260808 08:24:03.336586 7228 raft_consensus.cc:348] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.336872 7228 raft_consensus.cc:374] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:03.336932 7228 raft_consensus.cc:729] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7074660fb8b43a19681d85a5975468b, State: Initialized, Role: FOLLOWER
I20260808 08:24:03.336829 7227 raft_consensus.cc:348] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.337080 7227 raft_consensus.cc:374] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:03.337138 7227 raft_consensus.cc:729] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3aa8d9b82e54e35882e1b3c224f7bab, State: Initialized, Role: FOLLOWER
I20260808 08:24:03.337471 7228 consensus_queue.cc:260] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.337605 7227 consensus_queue.cc:260] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.338019 7226 ts_tablet_manager.cc:1428] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 08:24:03.338562 7228 ts_tablet_manager.cc:1428] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260808 08:24:03.338562 7227 ts_tablet_manager.cc:1428] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:24:03.338591 6827 heartbeater.cc:503] Master 127.12.13.62:45811 was elected leader, sending a full tablet report...
I20260808 08:24:03.338913 7202 heartbeater.cc:503] Master 127.12.13.62:45811 was elected leader, sending a full tablet report...
W20260808 08:24:03.344456 6830 tablet.cc:2369] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:24:03.371737 7233 raft_consensus.cc:482] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:24:03.371984 7233 raft_consensus.cc:504] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.373284 7233 leader_election.cc:288] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3aa8d9b82e54e35882e1b3c224f7bab (127.12.13.1:33489), 1d464b9d960e4a5e85e15fca59cc986a (127.12.13.2:43267)
I20260808 08:24:03.377549 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.373579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:33489 (local address 127.12.13.3:52565)
0808 08:24:03.373733 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.373751 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.373860 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.374292 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.374296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.374319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:03.374463 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.374471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.375404 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.375408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.376312 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.376322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.377100 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.377103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.377212 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.377249 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.377321 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.377399 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 08:24:03.377590 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.373681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:52565 (local address 127.12.13.1:33489)
0808 08:24:03.373818 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.373822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.373839 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:24:03.373998 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.374001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.374075 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.374273 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:03.374613 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.375277 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.376451 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.376967 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.377050 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.377101 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.377361 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.377439 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:24:03.377751 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.373978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:43267 (local address 127.12.13.3:47115)
0808 08:24:03.374357 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.374373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:03.374473 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:03.374717 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:03.374720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:03.374739 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:03.374863 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.374870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.375722 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.375724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.376594 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:03.376604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.377318 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.377324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:03.377424 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.377448 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:24:03.377527 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 08:24:03.377588 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":152,"threads_started":1}
I20260808 08:24:03.377777 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.374100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:47115 (local address 127.12.13.2:43267)
0808 08:24:03.374237 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.374243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.374469 (+ 226us) server_negotiation.cc:407] Connection header received
0808 08:24:03.374554 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.374557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.374612 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.374679 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:03.374975 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.375587 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.376719 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.377160 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.377264 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.377348 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.377562 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.377647 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 08:24:03.378294 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967cce9727f4163861d906157e671f1" candidate_uuid: "e7074660fb8b43a19681d85a5975468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" is_pre_election: true
I20260808 08:24:03.378388 6945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967cce9727f4163861d906157e671f1" candidate_uuid: "e7074660fb8b43a19681d85a5975468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d464b9d960e4a5e85e15fca59cc986a" is_pre_election: true
I20260808 08:24:03.378688 6945 raft_consensus.cc:2454] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7074660fb8b43a19681d85a5975468b in term 0.
I20260808 08:24:03.378687 6758 raft_consensus.cc:2454] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7074660fb8b43a19681d85a5975468b in term 0.
I20260808 08:24:03.379281 7046 leader_election.cc:302] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3aa8d9b82e54e35882e1b3c224f7bab, e7074660fb8b43a19681d85a5975468b; no voters:
I20260808 08:24:03.379587 7233 raft_consensus.cc:2781] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:24:03.379698 7233 raft_consensus.cc:482] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:24:03.379745 7233 raft_consensus.cc:3037] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:03.380836 7233 raft_consensus.cc:504] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.381258 7233 leader_election.cc:288] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [CANDIDATE]: Term 1 election: Requested vote from peers d3aa8d9b82e54e35882e1b3c224f7bab (127.12.13.1:33489), 1d464b9d960e4a5e85e15fca59cc986a (127.12.13.2:43267)
I20260808 08:24:03.381771 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967cce9727f4163861d906157e671f1" candidate_uuid: "e7074660fb8b43a19681d85a5975468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab"
I20260808 08:24:03.381832 6945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967cce9727f4163861d906157e671f1" candidate_uuid: "e7074660fb8b43a19681d85a5975468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d464b9d960e4a5e85e15fca59cc986a"
I20260808 08:24:03.381928 6758 raft_consensus.cc:3037] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:03.381968 6945 raft_consensus.cc:3037] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:03.383291 6945 raft_consensus.cc:2454] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7074660fb8b43a19681d85a5975468b in term 1.
I20260808 08:24:03.383291 6758 raft_consensus.cc:2454] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7074660fb8b43a19681d85a5975468b in term 1.
I20260808 08:24:03.383605 7046 leader_election.cc:302] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3aa8d9b82e54e35882e1b3c224f7bab, e7074660fb8b43a19681d85a5975468b; no voters:
I20260808 08:24:03.383812 7233 raft_consensus.cc:2781] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:03.384063 7233 raft_consensus.cc:686] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 1 LEADER]: Becoming Leader. State: Replica: e7074660fb8b43a19681d85a5975468b, State: Running, Role: LEADER
I20260808 08:24:03.384449 7233 consensus_queue.cc:237] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } }
I20260808 08:24:03.387759 6556 catalog_manager.cc:5495] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b reported cstate change: term changed from 0 to 1, leader changed from <none> to e7074660fb8b43a19681d85a5975468b (127.12.13.3). New cstate: current_term: 1 leader_uuid: "e7074660fb8b43a19681d85a5975468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7074660fb8b43a19681d85a5975468b" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 34461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 } health_report { overall_health: UNKNOWN } } }
I20260808 08:24:03.409970 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.406251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.12.13.3:34461)
0808 08:24:03.406428 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:03.406433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:03.406449 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:24:03.406617 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:03.406619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:03.406676 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:03.406798 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:03.407042 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.407548 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.408407 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:03.408621 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:03.408763 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:03.409074 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:03.409521 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:03.409660 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:03.409702 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 08:24:03.409754 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 08:24:03.416563 6758 raft_consensus.cc:1261] T 5967cce9727f4163861d906157e671f1 P d3aa8d9b82e54e35882e1b3c224f7bab [term 1 FOLLOWER]: Refusing update from remote peer e7074660fb8b43a19681d85a5975468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:03.416563 6945 raft_consensus.cc:1261] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 1 FOLLOWER]: Refusing update from remote peer e7074660fb8b43a19681d85a5975468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:03.417361 7233 consensus_queue.cc:1035] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3aa8d9b82e54e35882e1b3c224f7bab" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:03.417557 7239 consensus_queue.cc:1035] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d464b9d960e4a5e85e15fca59cc986a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 43267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:03.426724 7244 mvcc.cc:205] Tried to move back new op lower bound from 7316182808225710080 to 7316182808105357312. Current Snapshot: MvccSnapshot[applied={T|T < 7316182808225710080}]
I20260808 08:24:03.427240 7246 mvcc.cc:205] Tried to move back new op lower bound from 7316182808225710080 to 7316182808105357312. Current Snapshot: MvccSnapshot[applied={T|T < 7316182808225710080}]
I20260808 08:24:03.436091 6556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:03.436275 6556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:03.438884 6556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 165d595ee4d34a24b49192a7c1ae5178: Sending DeleteTablet for 3 replicas of tablet 5967cce9727f4163861d906157e671f1
I20260808 08:24:03.439738 7112 tablet_service.cc:1514] Processing DeleteTablet for tablet 5967cce9727f4163861d906157e671f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:03 UTC) from {username='slave'} at 127.0.0.1:36542
I20260808 08:24:03.439812 6922 tablet_service.cc:1514] Processing DeleteTablet for tablet 5967cce9727f4163861d906157e671f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:03 UTC) from {username='slave'} at 127.0.0.1:58780
I20260808 08:24:03.439872 6735 tablet_service.cc:1514] Processing DeleteTablet for tablet 5967cce9727f4163861d906157e671f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:03 UTC) from {username='slave'} at 127.0.0.1:51164
I20260808 08:24:03.440318 7257 tablet_replica.cc:330] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: stopping tablet replica
I20260808 08:24:03.440752 7257 raft_consensus.cc:2229] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:24:03.440760 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 6639
I20260808 08:24:03.441825 7257 raft_consensus.cc:2258] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:03.446352 7257 ts_tablet_manager.cc:1905] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:03.449676 7257 ts_tablet_manager.cc:1918] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:03.449776 7257 log.cc:1196] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/5967cce9727f4163861d906157e671f1
I20260808 08:24:03.450088 7257 ts_tablet_manager.cc:1939] T 5967cce9727f4163861d906157e671f1 P e7074660fb8b43a19681d85a5975468b: Deleting consensus metadata
I20260808 08:24:03.451406 6536 catalog_manager.cc:4841] TS e7074660fb8b43a19681d85a5975468b (127.12.13.3:34461): tablet 5967cce9727f4163861d906157e671f1 (table test-table [id=b4e2cfefc35f478c87453982bac81695]) successfully deleted
I20260808 08:24:03.451810 7258 tablet_replica.cc:330] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: stopping tablet replica
I20260808 08:24:03.452167 7258 raft_consensus.cc:2229] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:03.452518 7258 raft_consensus.cc:2258] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:03.454876 7258 ts_tablet_manager.cc:1905] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:03.458456 7258 ts_tablet_manager.cc:1918] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:03.458557 7258 log.cc:1196] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/5967cce9727f4163861d906157e671f1
I20260808 08:24:03.458838 7258 ts_tablet_manager.cc:1939] T 5967cce9727f4163861d906157e671f1 P 1d464b9d960e4a5e85e15fca59cc986a: Deleting consensus metadata
I20260808 08:24:03.463881 6534 catalog_manager.cc:4841] TS 1d464b9d960e4a5e85e15fca59cc986a (127.12.13.2:43267): tablet 5967cce9727f4163861d906157e671f1 (table test-table [id=b4e2cfefc35f478c87453982bac81695]) successfully deleted
W20260808 08:24:03.472996 6534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:33489 (error 108)
I20260808 08:24:03.473207 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 6832
I20260808 08:24:03.483248 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:03.482121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:33489 (local address 127.0.0.1:51168)
0808 08:24:03.482952 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:03.483093 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681}
W20260808 08:24:03.486750 6534 catalog_manager.cc:4584] TS d3aa8d9b82e54e35882e1b3c224f7bab (127.12.13.1:33489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5967cce9727f4163861d906157e671f1: Network error: Client connection negotiation failed: client connection to 127.12.13.1:33489: connect: Connection refused (error 111)
I20260808 08:24:03.497012 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 7018
I20260808 08:24:03.512428 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 6510
2026-08-08T08:24:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1149 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:03 dist-test-slave-6ntk krb5kdc[7263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:03 dist-test-slave-6ntk krb5kdc[7263](info): set up 2 sockets
Aug 08 08:24:03 dist-test-slave-6ntk krb5kdc[7263](info): commencing operation
krb5kdc: starting...
W20260808 08:24:04.751384 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:04 dist-test-slave-6ntk krb5kdc[7263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:04.893131 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:40839
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:05.022198 7284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:05.022511 7284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:05.022555 7284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:05.026628 7284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:05.026693 7284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:05.026713 7284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:05.026731 7284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:05.026746 7284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:05.032380 7284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:05.033510 7284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:05.034628 7284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:05.044116 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:05.044112 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:05.044152 7284 server_base.cc:1034] running on GCE node
W20260808 08:24:05.044111 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:05.044976 7284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:05.046262 7284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:05.047436 7284 hybrid_clock.cc:648] HybridClock initialized: now 1786177445047408 us; error 42 us; skew 500 ppm
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:05.050575 7284 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:05.051595 7284 webserver.cc:460] Webserver started at http://127.12.13.62:35481/ using document root <none> and password file <none>
I20260808 08:24:05.052171 7284 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:05.052232 7284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:05.052413 7284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:05.054606 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "137a0d1f61984d95bcbb018a32d36c83"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "4aad13f9b94480dbc008bc9005c882c3"
server_key_iv: "35fa074f90a181981a7ef426f24cf626"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.055141 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "137a0d1f61984d95bcbb018a32d36c83"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "4aad13f9b94480dbc008bc9005c882c3"
server_key_iv: "35fa074f90a181981a7ef426f24cf626"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.059201 7284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 08:24:05.061998 7305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.063859 7284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:24:05.063992 7284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "137a0d1f61984d95bcbb018a32d36c83"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "4aad13f9b94480dbc008bc9005c882c3"
server_key_iv: "35fa074f90a181981a7ef426f24cf626"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.064105 7284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:05.079515 7284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:05.083683 7284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:05.083878 7284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:05.092471 7284 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:40839
I20260808 08:24:05.092561 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:40839 every 8 connection(s)
I20260808 08:24:05.093585 7284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:05.096817 7372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:05.102172 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 7284
I20260808 08:24:05.102356 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:05.102664 12340 external_mini_cluster.cc:1428] Setting key 608739d3936eaaf1ea2296ba2fe2a8e9
I20260808 08:24:05.103886 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83: Bootstrap starting.
I20260808 08:24:05.106899 7372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:05.107796 7372 log.cc:824] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:05.110354 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83: No bootstrap required, opened a new log
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:05.115087 7372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137a0d1f61984d95bcbb018a32d36c83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40839 } }
I20260808 08:24:05.115432 7372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:05.115494 7372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137a0d1f61984d95bcbb018a32d36c83, State: Initialized, Role: FOLLOWER
I20260808 08:24:05.116111 7372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137a0d1f61984d95bcbb018a32d36c83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40839 } }
I20260808 08:24:05.116287 7372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:05.116358 7372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:05.116468 7372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:05.117730 7372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137a0d1f61984d95bcbb018a32d36c83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40839 } }
I20260808 08:24:05.118161 7372 leader_election.cc:302] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 137a0d1f61984d95bcbb018a32d36c83; no voters:
I20260808 08:24:05.118556 7372 leader_election.cc:288] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:05.118695 7379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:05.118978 7379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [term 1 LEADER]: Becoming Leader. State: Replica: 137a0d1f61984d95bcbb018a32d36c83, State: Running, Role: LEADER
I20260808 08:24:05.119338 7379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137a0d1f61984d95bcbb018a32d36c83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40839 } }
I20260808 08:24:05.120064 7372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:05.121042 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.104331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.12.13.62:40839)
0808 08:24:05.104818 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.104838 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.104883 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:24:05.105786 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.105813 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.106328 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.106722 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:05.107862 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.109228 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.110308 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.111021 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.111204 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.114394 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:05.114401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:05.114421 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:05.114457 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:05.116837 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.117343 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.117353 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.117363 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.117528 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.118006 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.118009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.118011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.118610 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:05.118837 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:05.119650 (+ 813us) server_negotiation.cc:299] Negotiation successful
0808 08:24:05.119973 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
I20260808 08:24:05.121408 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 137a0d1f61984d95bcbb018a32d36c83. Latest consensus state: current_term: 1 leader_uuid: "137a0d1f61984d95bcbb018a32d36c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137a0d1f61984d95bcbb018a32d36c83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40839 } } }
I20260808 08:24:05.121526 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:05.122017 7392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:05.122119 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "137a0d1f61984d95bcbb018a32d36c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137a0d1f61984d95bcbb018a32d36c83" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 40839 } } }
I20260808 08:24:05.122210 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:05.125459 7392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:05.132129 7392 catalog_manager.cc:1347] Generated new cluster ID: 2223210c26564fd496e26139229bb1b5
I20260808 08:24:05.132342 7392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:05.142499 7392 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:05.143637 7392 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:05.148669 7392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83: Generated new TSK 0
I20260808 08:24:05.149470 7392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:24:05.193749 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40839
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:05.346822 7415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:05.347108 7415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:05.347153 7415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:05.351277 7415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:05.351351 7415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:05.351413 7415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:24:05.356860 7415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:05.358076 7415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:05.359149 7415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:05.370381 7425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:05.370306 7420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:05.370584 7415 server_base.cc:1034] running on GCE node
W20260808 08:24:05.370762 7421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:05.371236 7415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:05.372004 7415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:05.373216 7415 hybrid_clock.cc:648] HybridClock initialized: now 1786177445373196 us; error 38 us; skew 500 ppm
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:05.376250 7415 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:24:05.377140 7415 webserver.cc:460] Webserver started at http://127.12.13.1:41195/ using document root <none> and password file <none>
I20260808 08:24:05.377699 7415 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:05.377781 7415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:05.378058 7415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:05.380173 7415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "b71a3ab3b0e64633bce3785f55b5ba96"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "cd619cbc75991c820467ee26a98635da"
server_key_iv: "05122519cdc55ea3c90b8835b26f3d86"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.380692 7415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "b71a3ab3b0e64633bce3785f55b5ba96"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "cd619cbc75991c820467ee26a98635da"
server_key_iv: "05122519cdc55ea3c90b8835b26f3d86"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.384593 7415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:05.387248 7436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.388607 7415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 08:24:05.388751 7415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "b71a3ab3b0e64633bce3785f55b5ba96"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "cd619cbc75991c820467ee26a98635da"
server_key_iv: "05122519cdc55ea3c90b8835b26f3d86"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.388877 7415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:05.432869 7415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:05.436210 7415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:05.436448 7415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:05.437155 7415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:05.438151 7415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:05.438205 7415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.438247 7415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:05.438304 7415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.447890 7415 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:41997
I20260808 08:24:05.447957 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:41997 every 8 connection(s)
I20260808 08:24:05.448953 7415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:24:05.454641 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 7415
I20260808 08:24:05.454759 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:24:05.455114 12340 external_mini_cluster.cc:1428] Setting key e74bb6965fb336a82e4dc40c83ac1ff0
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:05.464957 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.451071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:57815 (local address 127.12.13.62:40839)
0808 08:24:05.451230 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.451234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.452111 (+ 877us) server_negotiation.cc:407] Connection header received
0808 08:24:05.453620 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.453623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.453689 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.453815 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:05.455328 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.455902 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.457147 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.457383 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.457498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.460460 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:05.460466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:05.460472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:05.460507 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:05.462394 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.463029 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.463034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.463036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.463138 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.463520 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.463524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.463529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.463838 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:05.463964 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:05.464584 (+ 620us) server_negotiation.cc:299] Negotiation successful
0808 08:24:05.464738 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 08:24:05.465956 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.451451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:40839 (local address 127.12.13.1:57815)
0808 08:24:05.451938 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:05.451986 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:05.453351 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:05.454034 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:05.454041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:05.454447 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:05.455092 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.455108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.456050 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.456056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:05.456997 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.457007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.457558 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.457562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:05.457739 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.458505 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:05.458523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:05.460271 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:05.462604 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:05.462609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:05.462623 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:05.462913 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:05.463241 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:05.463243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:05.463244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:05.463406 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:05.463994 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:05.464000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:05.464302 (+ 302us) client_negotiation.cc:769] Sending connection context
0808 08:24:05.464488 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 08:24:05.464800 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:24:05.467557 7590 heartbeater.cc:347] Connected to a master server at 127.12.13.62:40839
I20260808 08:24:05.467878 7590 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:05.468501 7590 heartbeater.cc:511] Master 127.12.13.62:40839 requested a full tablet report, sending...
I20260808 08:24:05.470259 7320 ts_manager.cc:194] Registered new tserver with Master: b71a3ab3b0e64633bce3785f55b5ba96 (127.12.13.1:41997)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
I20260808 08:24:05.471766 7320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:57815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:24:05.489679 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40839
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:05.608888 7597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:05.609169 7597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:05.609215 7597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:05.613283 7597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:05.613353 7597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:05.613409 7597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:24:05.618836 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:05.620061 7597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:05.621268 7597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:05.632380 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:05.632330 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:05.632395 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:05.632584 7597 server_base.cc:1034] running on GCE node
I20260808 08:24:05.633090 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:05.633854 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:05.635135 7597 hybrid_clock.cc:648] HybridClock initialized: now 1786177445635098 us; error 51 us; skew 500 ppm
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:05.638652 7597 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:24:05.639746 7597 webserver.cc:460] Webserver started at http://127.12.13.2:37219/ using document root <none> and password file <none>
I20260808 08:24:05.640466 7597 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:05.640569 7597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:05.640812 7597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:05.643072 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "31450f4e18374de7a41daf7816812b74"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "f4eb2292d20a8f9d7c47fa874864886c"
server_key_iv: "0ea1b7a87de7c4a1e0c0bd007372bb3f"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.643623 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "31450f4e18374de7a41daf7816812b74"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "f4eb2292d20a8f9d7c47fa874864886c"
server_key_iv: "0ea1b7a87de7c4a1e0c0bd007372bb3f"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.647724 7597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:05.650889 7618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.652352 7597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:24:05.652519 7597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "31450f4e18374de7a41daf7816812b74"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "f4eb2292d20a8f9d7c47fa874864886c"
server_key_iv: "0ea1b7a87de7c4a1e0c0bd007372bb3f"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.652652 7597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:05.672071 7597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:05.675935 7597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:05.676198 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:05.676921 7597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:05.677843 7597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:05.677966 7597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.678038 7597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:05.678097 7597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.687875 7597 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:46591
I20260808 08:24:05.687901 7772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:46591 every 8 connection(s)
I20260808 08:24:05.688876 7597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:24:05.698643 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 7597
I20260808 08:24:05.698834 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:05.699170 12340 external_mini_cluster.cc:1428] Setting key dec108b8f820a5b7566dd0ad624ea246
I20260808 08:24:05.704155 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.690962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:44875 (local address 127.12.13.62:40839)
0808 08:24:05.691126 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.691131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.692645 (+ 1514us) server_negotiation.cc:407] Connection header received
0808 08:24:05.693601 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.693605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.693666 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.693742 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:05.695061 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.695590 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.696727 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.696948 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.697030 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.699807 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:05.699811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:05.699813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:05.699839 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:05.701508 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.702244 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.702246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.702248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.702331 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.702807 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.702811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.702813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.703114 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:05.703228 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:05.703794 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 08:24:05.703949 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:24:05.704968 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.691460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:40839 (local address 127.12.13.2:44875)
0808 08:24:05.692454 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:05.692517 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:05.693350 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:05.693990 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:05.693998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:05.694409 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:05.694912 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.694925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.695709 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.695712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:05.696609 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.696618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.697094 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.697097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:05.697244 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.697855 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:05.697937 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:05.699608 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:05.701643 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:05.701653 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:05.701675 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:05.702145 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:05.702434 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:05.702440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:05.702444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:05.702685 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:05.703237 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:05.703243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:05.703525 (+ 282us) client_negotiation.cc:769] Sending connection context
0808 08:24:05.703686 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 08:24:05.703977 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":558,"threads_started":1}
I20260808 08:24:05.706735 7773 heartbeater.cc:347] Connected to a master server at 127.12.13.62:40839
I20260808 08:24:05.707026 7773 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:05.707577 7773 heartbeater.cc:511] Master 127.12.13.62:40839 requested a full tablet report, sending...
I20260808 08:24:05.708750 7320 ts_manager.cc:194] Registered new tserver with Master: 31450f4e18374de7a41daf7816812b74 (127.12.13.2:46591)
I20260808 08:24:05.709355 7320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:44875
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:24:05.733966 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40839
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:05.855954 7779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:05.856305 7779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:05.856407 7779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:05.860682 7779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:05.860786 7779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:05.860882 7779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:24:05.866310 7779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:05.867532 7779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:05.868760 7779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:05.879660 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:05.879783 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:05.879712 7789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:05.880050 7779 server_base.cc:1034] running on GCE node
I20260808 08:24:05.880507 7779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:05.881295 7779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:05.882535 7779 hybrid_clock.cc:648] HybridClock initialized: now 1786177445882520 us; error 30 us; skew 500 ppm
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:05.885556 7779 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:24:05.886590 7779 webserver.cc:460] Webserver started at http://127.12.13.3:41863/ using document root <none> and password file <none>
I20260808 08:24:05.887265 7779 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:05.887329 7779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:05.887589 7779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:05.889750 7779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "e5e6c363230d41ff887184f038a9c839"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "187dc5b6629d54079d0728cf2dda9459"
server_key_iv: "a202a3f05362cba238be31495c3e20b3"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.890429 7779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "e5e6c363230d41ff887184f038a9c839"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "187dc5b6629d54079d0728cf2dda9459"
server_key_iv: "a202a3f05362cba238be31495c3e20b3"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.894695 7779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:05.897356 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.898723 7779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:05.898881 7779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "e5e6c363230d41ff887184f038a9c839"
format_stamp: "Formatted at 2026-08-08 08:24:05 on dist-test-slave-6ntk"
server_key: "187dc5b6629d54079d0728cf2dda9459"
server_key_iv: "a202a3f05362cba238be31495c3e20b3"
server_key_version: "encryptionkey@0"
I20260808 08:24:05.898996 7779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:05.925374 7779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:05.928764 7779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:05.929023 7779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:05.929708 7779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:05.930706 7779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:05.930761 7779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.930804 7779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:05.930819 7779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:05.940711 7779 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:41023
I20260808 08:24:05.940779 7952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:41023 every 8 connection(s)
I20260808 08:24:05.941713 7779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:24:05.942488 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 7779
I20260808 08:24:05.942632 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:24:05.942958 12340 external_mini_cluster.cc:1428] Setting key 3257ef9c48b77e2db72d02e507f0be73
Aug 08 08:24:05 dist-test-slave-6ntk krb5kdc[7263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:05.956552 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.943908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:43475 (local address 127.12.13.62:40839)
0808 08:24:05.944160 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.944164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.945119 (+ 955us) server_negotiation.cc:407] Connection header received
0808 08:24:05.946654 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.946657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.946704 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.946773 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:05.947990 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.948522 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.949663 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.949916 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.950037 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.952560 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:05.952563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:05.952565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:05.952592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:05.954367 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.954862 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.954865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.954866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.954947 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.955271 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.955273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.955274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.955482 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:05.955588 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:05.956245 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 08:24:05.956376 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260808 08:24:05.957739 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.944417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:40839 (local address 127.12.13.3:43475)
0808 08:24:05.944930 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:05.944990 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:05.946453 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:05.946940 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:05.946947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:05.947321 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:05.947855 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.947869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.948641 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.948643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:05.949544 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.949553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.950070 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.950073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:05.950220 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.950826 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:05.950842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:05.952370 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:05.954484 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:05.954489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:05.954501 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:05.954760 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:05.955038 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:05.955040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:05.955042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:05.955183 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:05.955619 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:05.955626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:05.955956 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 08:24:05.956199 (+ 243us) client_negotiation.cc:240] Negotiation successful
0808 08:24:05.956570 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260808 08:24:05.958951 7953 heartbeater.cc:347] Connected to a master server at 127.12.13.62:40839
I20260808 08:24:05.959223 7953 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:05.959753 7953 heartbeater.cc:511] Master 127.12.13.62:40839 requested a full tablet report, sending...
I20260808 08:24:05.960875 7320 ts_manager.cc:194] Registered new tserver with Master: e5e6c363230d41ff887184f038a9c839 (127.12.13.3:41023)
I20260808 08:24:05.961515 7320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:43475
I20260808 08:24:05.970177 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:24:05.981102 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.972554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.12.13.62:40839)
0808 08:24:05.972742 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.972746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.972834 (+ 88us) server_negotiation.cc:407] Connection header received
0808 08:24:05.973022 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.973025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.973074 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.973150 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:05.974093 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.974743 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.975729 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.975945 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:05.976030 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:05.977843 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:05.977846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:05.977848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:05.977976 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:05.979318 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.979822 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.979824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.979826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.979900 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:05.980177 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:05.980180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:05.980181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:05.980438 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:05.980558 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:05.980801 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 08:24:05.980885 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 08:24:05.984551 7320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:24:05.987761 7320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:24:06.003650 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.996912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:46591 (local address 127.0.0.1:33480)
0808 08:24:05.997946 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:05.997961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:05.998040 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:05.998718 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:05.998721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:05.998765 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:05.998857 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.998864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.000240 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.000243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.001685 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.001695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.003004 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.003008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.003123 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.003185 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.003331 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.003477 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":2,"thread_start_us":117,"threads_started":1}
I20260808 08:24:06.004101 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.997115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:41997 (local address 127.0.0.1:46340)
0808 08:24:05.998162 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:05.998176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:05.998267 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:05.998961 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:05.998963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:05.998982 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:05.999078 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.999085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.000033 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.000038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.002199 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.002211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.003683 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.003687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.003805 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.003831 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.003903 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.003968 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":858,"thread_start_us":83,"threads_started":1}
I20260808 08:24:06.004127 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.997553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.12.13.2:46591)
0808 08:24:05.997972 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.997987 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.998018 (+ 31us) server_negotiation.cc:407] Connection header received
0808 08:24:05.998192 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.998197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.998419 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.998566 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:05.998993 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.000106 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.001835 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.002675 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.002780 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.002936 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.003846 (+ 910us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.003953 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":100,"threads_started":1}
I20260808 08:24:06.004588 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.997553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.12.13.1:41997)
0808 08:24:05.998515 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.998521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.998541 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:24:05.998615 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.998620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.998836 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.998989 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:05.999178 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:05.999905 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.002319 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.003223 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.003480 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.003623 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.004336 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.004445 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":68,"threads_started":1}
I20260808 08:24:06.004583 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.996731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:41023 (local address 127.0.0.1:59354)
0808 08:24:05.997545 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:05.997599 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:05.997735 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:05.998513 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:05.998519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:05.998544 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:05.998677 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:05.998683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.000202 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.000205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.002775 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.002784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.004161 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.004168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.004257 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.004279 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.004364 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.004440 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":1529,"thread_start_us":94,"threads_started":1}
I20260808 08:24:06.004637 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:05.996869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.12.13.3:41023)
0808 08:24:05.997522 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:05.997540 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:05.997934 (+ 394us) server_negotiation.cc:407] Connection header received
0808 08:24:05.998031 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:05.998037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:05.998255 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:05.998414 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:05.998823 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.000052 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.002926 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.003815 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.003948 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.004094 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.004398 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.004489 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":161,"threads_started":1}
I20260808 08:24:06.007478 7864 tablet_service.cc:1467] Processing CreateTablet for tablet 6c44fc3ed05646deacb73c4a069f18a7 (DEFAULT_TABLE table=test-table [id=64098c1fef0d4d8bb9940210e66108f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:06.007541 7502 tablet_service.cc:1467] Processing CreateTablet for tablet 6c44fc3ed05646deacb73c4a069f18a7 (DEFAULT_TABLE table=test-table [id=64098c1fef0d4d8bb9940210e66108f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:06.007543 7684 tablet_service.cc:1467] Processing CreateTablet for tablet 6c44fc3ed05646deacb73c4a069f18a7 (DEFAULT_TABLE table=test-table [id=64098c1fef0d4d8bb9940210e66108f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:06.008571 7864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c44fc3ed05646deacb73c4a069f18a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:06.008589 7502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c44fc3ed05646deacb73c4a069f18a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:06.008590 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c44fc3ed05646deacb73c4a069f18a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:06.015554 7979 tablet_bootstrap.cc:492] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Bootstrap starting.
I20260808 08:24:06.016642 7981 tablet_bootstrap.cc:492] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Bootstrap starting.
I20260808 08:24:06.018157 7979 tablet_bootstrap.cc:654] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:06.018236 7980 tablet_bootstrap.cc:492] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96: Bootstrap starting.
I20260808 08:24:06.019042 7981 tablet_bootstrap.cc:654] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:06.019381 7979 log.cc:824] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:06.019919 7981 log.cc:824] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:06.020566 7980 tablet_bootstrap.cc:654] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:06.021435 7980 log.cc:824] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:06.022085 7981 tablet_bootstrap.cc:492] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: No bootstrap required, opened a new log
I20260808 08:24:06.022086 7979 tablet_bootstrap.cc:492] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: No bootstrap required, opened a new log
I20260808 08:24:06.022293 7979 ts_tablet_manager.cc:1397] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260808 08:24:06.022305 7981 ts_tablet_manager.cc:1397] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:24:06.023276 7980 tablet_bootstrap.cc:492] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96: No bootstrap required, opened a new log
I20260808 08:24:06.023488 7980 ts_tablet_manager.cc:1397] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 08:24:06.026396 7979 raft_consensus.cc:348] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.026396 7981 raft_consensus.cc:348] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.026696 7981 raft_consensus.cc:374] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:06.026696 7979 raft_consensus.cc:374] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:06.026759 7981 raft_consensus.cc:729] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31450f4e18374de7a41daf7816812b74, State: Initialized, Role: FOLLOWER
I20260808 08:24:06.026759 7979 raft_consensus.cc:729] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e6c363230d41ff887184f038a9c839, State: Initialized, Role: FOLLOWER
I20260808 08:24:06.027079 7980 raft_consensus.cc:348] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.027263 7979 consensus_queue.cc:260] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.027247 7981 consensus_queue.cc:260] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.027369 7980 raft_consensus.cc:374] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:06.027436 7980 raft_consensus.cc:729] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71a3ab3b0e64633bce3785f55b5ba96, State: Initialized, Role: FOLLOWER
I20260808 08:24:06.027900 7980 consensus_queue.cc:260] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.028319 7981 ts_tablet_manager.cc:1428] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 08:24:06.028309 7979 ts_tablet_manager.cc:1428] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 08:24:06.028671 7953 heartbeater.cc:503] Master 127.12.13.62:40839 was elected leader, sending a full tablet report...
I20260808 08:24:06.028671 7773 heartbeater.cc:503] Master 127.12.13.62:40839 was elected leader, sending a full tablet report...
I20260808 08:24:06.028955 7980 ts_tablet_manager.cc:1428] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 08:24:06.029775 7590 heartbeater.cc:503] Master 127.12.13.62:40839 was elected leader, sending a full tablet report...
I20260808 08:24:06.089704 7987 raft_consensus.cc:482] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:24:06.090026 7987 raft_consensus.cc:504] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.091423 7987 leader_election.cc:288] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31450f4e18374de7a41daf7816812b74 (127.12.13.2:46591), b71a3ab3b0e64633bce3785f55b5ba96 (127.12.13.1:41997)
I20260808 08:24:06.096352 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.091694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:41997 (local address 127.12.13.3:55241)
0808 08:24:06.091936 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:06.091953 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:06.092062 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:06.092435 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:06.092439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:06.092454 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:06.092605 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.092611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.093593 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.093598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.094861 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.094873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.095891 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.095894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.096005 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.096034 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.096107 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.096186 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 08:24:06.096557 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.091823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:55671 (local address 127.12.13.2:46591)
0808 08:24:06.091977 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:06.091983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:06.092280 (+ 297us) server_negotiation.cc:407] Connection header received
0808 08:24:06.092615 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:06.092619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:06.092673 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:06.092749 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:06.093083 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.093803 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.095269 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.095879 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.095955 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.096014 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.096340 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.096428 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 08:24:06.096665 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.091902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:55241 (local address 127.12.13.1:41997)
0808 08:24:06.092034 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:06.092038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:06.092052 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:24:06.092251 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:06.092254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:06.092311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:06.092400 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:06.092731 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.093475 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.095004 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.095643 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.095743 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.095826 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.096434 (+ 608us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.096546 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:24:06.097079 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.091696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:46591 (local address 127.12.13.3:55671)
0808 08:24:06.092178 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:06.092195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:06.092435 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:06.092808 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:06.092811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:06.092830 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:06.092945 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.092951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.093971 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.093975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.095134 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.095143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.096105 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.096108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.096199 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.096218 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.096299 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.096352 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":66,"thread_start_us":154,"threads_started":1}
I20260808 08:24:06.097326 7526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c44fc3ed05646deacb73c4a069f18a7" candidate_uuid: "e5e6c363230d41ff887184f038a9c839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" is_pre_election: true
I20260808 08:24:06.097653 7526 raft_consensus.cc:2454] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6c363230d41ff887184f038a9c839 in term 0.
I20260808 08:24:06.097810 7711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c44fc3ed05646deacb73c4a069f18a7" candidate_uuid: "e5e6c363230d41ff887184f038a9c839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31450f4e18374de7a41daf7816812b74" is_pre_election: true
I20260808 08:24:06.098153 7711 raft_consensus.cc:2454] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6c363230d41ff887184f038a9c839 in term 0.
I20260808 08:24:06.098209 7801 leader_election.cc:302] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b71a3ab3b0e64633bce3785f55b5ba96, e5e6c363230d41ff887184f038a9c839; no voters:
I20260808 08:24:06.098491 7987 raft_consensus.cc:2781] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:24:06.098613 7987 raft_consensus.cc:482] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:24:06.098671 7987 raft_consensus.cc:3037] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:06.099812 7987 raft_consensus.cc:504] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.100219 7987 leader_election.cc:288] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [CANDIDATE]: Term 1 election: Requested vote from peers 31450f4e18374de7a41daf7816812b74 (127.12.13.2:46591), b71a3ab3b0e64633bce3785f55b5ba96 (127.12.13.1:41997)
I20260808 08:24:06.100628 7711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c44fc3ed05646deacb73c4a069f18a7" candidate_uuid: "e5e6c363230d41ff887184f038a9c839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31450f4e18374de7a41daf7816812b74"
I20260808 08:24:06.100628 7526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c44fc3ed05646deacb73c4a069f18a7" candidate_uuid: "e5e6c363230d41ff887184f038a9c839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b71a3ab3b0e64633bce3785f55b5ba96"
I20260808 08:24:06.100768 7711 raft_consensus.cc:3037] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:06.100768 7526 raft_consensus.cc:3037] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:06.102269 7526 raft_consensus.cc:2454] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6c363230d41ff887184f038a9c839 in term 1.
I20260808 08:24:06.102269 7711 raft_consensus.cc:2454] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6c363230d41ff887184f038a9c839 in term 1.
I20260808 08:24:06.102632 7805 leader_election.cc:302] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31450f4e18374de7a41daf7816812b74, e5e6c363230d41ff887184f038a9c839; no voters:
I20260808 08:24:06.102799 7987 raft_consensus.cc:2781] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:06.103138 7987 raft_consensus.cc:686] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 1 LEADER]: Becoming Leader. State: Replica: e5e6c363230d41ff887184f038a9c839, State: Running, Role: LEADER
I20260808 08:24:06.103638 7987 consensus_queue.cc:237] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } }
I20260808 08:24:06.107445 7326 catalog_manager.cc:5495] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e6c363230d41ff887184f038a9c839 (127.12.13.3). New cstate: current_term: 1 leader_uuid: "e5e6c363230d41ff887184f038a9c839" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e6c363230d41ff887184f038a9c839" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 41023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 } health_report { overall_health: UNKNOWN } } }
I20260808 08:24:06.129654 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.125473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.12.13.3:41023)
0808 08:24:06.125642 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:06.125650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:06.125670 (+ 20us) server_negotiation.cc:407] Connection header received
0808 08:24:06.125785 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:06.125788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:06.125850 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:06.126033 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:06.126344 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.126923 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.127916 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.128165 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.128299 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.128667 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:06.129130 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:06.129324 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.129440 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.129492 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 08:24:06.136526 7711 raft_consensus.cc:1261] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 1 FOLLOWER]: Refusing update from remote peer e5e6c363230d41ff887184f038a9c839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:06.136524 7526 raft_consensus.cc:1261] T 6c44fc3ed05646deacb73c4a069f18a7 P b71a3ab3b0e64633bce3785f55b5ba96 [term 1 FOLLOWER]: Refusing update from remote peer e5e6c363230d41ff887184f038a9c839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:06.137365 7987 consensus_queue.cc:1035] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b71a3ab3b0e64633bce3785f55b5ba96" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:06.137564 7990 consensus_queue.cc:1035] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31450f4e18374de7a41daf7816812b74" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:06.144716 7993 mvcc.cc:205] Tried to move back new op lower bound from 7316182819367215104 to 7316182819243720704. Current Snapshot: MvccSnapshot[applied={T|T < 7316182819367215104}]
I20260808 08:24:06.156570 7324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:06.156797 7324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:06.159763 7324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 137a0d1f61984d95bcbb018a32d36c83: Sending DeleteTablet for 3 replicas of tablet 6c44fc3ed05646deacb73c4a069f18a7
I20260808 08:24:06.160640 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c44fc3ed05646deacb73c4a069f18a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:06 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:33480
I20260808 08:24:06.160841 7502 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c44fc3ed05646deacb73c4a069f18a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:06 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:46340
I20260808 08:24:06.160907 7864 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c44fc3ed05646deacb73c4a069f18a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:06 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:59354
I20260808 08:24:06.161361 8010 tablet_replica.cc:330] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: stopping tablet replica
I20260808 08:24:06.161386 8008 tablet_replica.cc:330] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: stopping tablet replica
I20260808 08:24:06.161736 8008 raft_consensus.cc:2229] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:06.161736 8010 raft_consensus.cc:2229] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:24:06.162052 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 7415
I20260808 08:24:06.165092 8008 raft_consensus.cc:2258] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:06.165369 8010 raft_consensus.cc:2258] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:06.168793 8010 ts_tablet_manager.cc:1905] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:06.170724 8008 ts_tablet_manager.cc:1905] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:06.172179 8010 ts_tablet_manager.cc:1918] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:06.172403 8010 log.cc:1196] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/6c44fc3ed05646deacb73c4a069f18a7
I20260808 08:24:06.172883 8010 ts_tablet_manager.cc:1939] T 6c44fc3ed05646deacb73c4a069f18a7 P e5e6c363230d41ff887184f038a9c839: Deleting consensus metadata
I20260808 08:24:06.173732 8008 ts_tablet_manager.cc:1918] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:06.173847 8008 log.cc:1196] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/6c44fc3ed05646deacb73c4a069f18a7
I20260808 08:24:06.174203 8008 ts_tablet_manager.cc:1939] T 6c44fc3ed05646deacb73c4a069f18a7 P 31450f4e18374de7a41daf7816812b74: Deleting consensus metadata
I20260808 08:24:06.174728 7306 catalog_manager.cc:4841] TS e5e6c363230d41ff887184f038a9c839 (127.12.13.3:41023): tablet 6c44fc3ed05646deacb73c4a069f18a7 (table test-table [id=64098c1fef0d4d8bb9940210e66108f4]) successfully deleted
I20260808 08:24:06.175561 7311 catalog_manager.cc:4841] TS 31450f4e18374de7a41daf7816812b74 (127.12.13.2:46591): tablet 6c44fc3ed05646deacb73c4a069f18a7 (table test-table [id=64098c1fef0d4d8bb9940210e66108f4]) successfully deleted
W20260808 08:24:06.178464 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:41997 (error 108)
I20260808 08:24:06.179409 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 7597
I20260808 08:24:06.179651 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.179266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:41997 (local address 127.0.0.1:46342)
0808 08:24:06.179481 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:06.179555 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:41997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260808 08:24:06.193610 7308 catalog_manager.cc:4584] TS b71a3ab3b0e64633bce3785f55b5ba96 (127.12.13.1:41997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c44fc3ed05646deacb73c4a069f18a7: Network error: Client connection negotiation failed: client connection to 127.12.13.1:41997: connect: Connection refused (error 111)
I20260808 08:24:06.203899 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 7779
I20260808 08:24:06.219585 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 7284
2026-08-08T08:24:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2715 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T08:24:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:06Z Disabled control of system clock
I20260808 08:24:06.265762 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36355
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 08:24:06.408039 8015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:06.408336 8015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:06.408375 8015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:06.412396 8015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:06.412456 8015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:06.412472 8015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:06.412489 8015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:06.412504 8015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:06.417263 8015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:06.418366 8015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:06.419752 8015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:06.428503 8024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:06.428519 8021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:06.428529 8022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:06.428957 8015 server_base.cc:1034] running on GCE node
I20260808 08:24:06.429564 8015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:06.430807 8015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:06.432009 8015 hybrid_clock.cc:648] HybridClock initialized: now 1786177446431975 us; error 49 us; skew 500 ppm
I20260808 08:24:06.434003 8015 webserver.cc:460] Webserver started at http://127.12.13.62:45861/ using document root <none> and password file <none>
I20260808 08:24:06.434629 8015 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:06.434731 8015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:06.434981 8015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:06.437089 8015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "b9604168c04a4c1f9b49c9fc9e23653b"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "b270559942e9bcea45452e55974946eb"
server_key_iv: "b01187dac561479f61a4cf794ce26aa5"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.437668 8015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "b9604168c04a4c1f9b49c9fc9e23653b"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "b270559942e9bcea45452e55974946eb"
server_key_iv: "b01187dac561479f61a4cf794ce26aa5"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.442265 8015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 08:24:06.444888 8033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.446383 8015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:06.446530 8015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "b9604168c04a4c1f9b49c9fc9e23653b"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "b270559942e9bcea45452e55974946eb"
server_key_iv: "b01187dac561479f61a4cf794ce26aa5"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.446652 8015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:06.464838 8015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:06.465633 8015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:06.465842 8015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:06.473776 8015 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36355
I20260808 08:24:06.473824 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36355 every 8 connection(s)
I20260808 08:24:06.474867 8015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:06.478396 8112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:06.483239 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 8015
I20260808 08:24:06.483362 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:06.483664 12340 external_mini_cluster.cc:1428] Setting key 985a7fb368c396c06f6f047fbd636cc1
I20260808 08:24:06.484946 8112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b: Bootstrap starting.
I20260808 08:24:06.487365 8112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:06.488297 8112 log.cc:824] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:06.490705 8112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b: No bootstrap required, opened a new log
I20260808 08:24:06.493747 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.485291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.12.13.62:36355)
0808 08:24:06.486029 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:06.486048 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:06.486086 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:24:06.486760 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:06.486779 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:06.487193 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:06.487529 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:06.487927 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.489155 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.490386 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.490913 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.491032 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.491930 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:06.491935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:06.491949 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:06.492345 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:06.492402 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:06.492423 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:06.492500 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.492674 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.492979 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260808 08:24:06.494889 8112 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36355 } }
I20260808 08:24:06.495114 8112 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:06.495187 8112 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9604168c04a4c1f9b49c9fc9e23653b, State: Initialized, Role: FOLLOWER
I20260808 08:24:06.495656 8112 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36355 } }
I20260808 08:24:06.495779 8112 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:06.495878 8112 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:06.496023 8112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:06.497025 8112 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36355 } }
I20260808 08:24:06.497362 8112 leader_election.cc:302] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9604168c04a4c1f9b49c9fc9e23653b; no voters:
I20260808 08:24:06.497658 8112 leader_election.cc:288] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:06.497814 8119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:06.498101 8119 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [term 1 LEADER]: Becoming Leader. State: Replica: b9604168c04a4c1f9b49c9fc9e23653b, State: Running, Role: LEADER
I20260808 08:24:06.498433 8119 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36355 } }
I20260808 08:24:06.499145 8112 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:06.500388 8121 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9604168c04a4c1f9b49c9fc9e23653b. Latest consensus state: current_term: 1 leader_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36355 } } }
I20260808 08:24:06.500545 8121 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:06.500388 8120 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9604168c04a4c1f9b49c9fc9e23653b" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36355 } } }
I20260808 08:24:06.500699 8120 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:06.501147 8137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:06.504227 8137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:06.509955 8137 catalog_manager.cc:1347] Generated new cluster ID: e45773e4c58a47519f40dd56711bc9e0
I20260808 08:24:06.510032 8137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:06.524950 8137 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:06.526044 8137 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:06.536621 8137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b: Generated new TSK 0
I20260808 08:24:06.537437 8137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:06.546801 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36355
--builtin_ntp_servers=127.12.13.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 08:24:06.675392 8149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:06.675710 8149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:06.675756 8149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:06.680042 8149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:06.680126 8149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:06.680186 8149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:24:06.685379 8149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:06.686689 8149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:06.688215 8149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:06.698802 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:06.698750 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:06.698750 8159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:06.699209 8149 server_base.cc:1034] running on GCE node
I20260808 08:24:06.699724 8149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:06.700556 8149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:06.701767 8149 hybrid_clock.cc:648] HybridClock initialized: now 1786177446701754 us; error 35 us; skew 500 ppm
I20260808 08:24:06.703982 8149 webserver.cc:460] Webserver started at http://127.12.13.1:37117/ using document root <none> and password file <none>
I20260808 08:24:06.704887 8149 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:06.704996 8149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:06.705228 8149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:06.707549 8149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "8a21097ff79b482fad71c4ec3160302c"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "a0700ff3a7c1ce2460d736aae633dea7"
server_key_iv: "9793d2d3925fab53cbf42ead4d67aa15"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.708174 8149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "8a21097ff79b482fad71c4ec3160302c"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "a0700ff3a7c1ce2460d736aae633dea7"
server_key_iv: "9793d2d3925fab53cbf42ead4d67aa15"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.712212 8149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:06.715157 8167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.716624 8149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:06.716789 8149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "8a21097ff79b482fad71c4ec3160302c"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "a0700ff3a7c1ce2460d736aae633dea7"
server_key_iv: "9793d2d3925fab53cbf42ead4d67aa15"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.716929 8149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:06.744626 8149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:06.745558 8149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:06.745769 8149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:06.746519 8149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:06.747536 8149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:06.747586 8149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.747627 8149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:06.747643 8149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.757284 8149 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:43045
I20260808 08:24:06.757395 8334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:43045 every 8 connection(s)
I20260808 08:24:06.758461 8149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:24:06.766031 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 8149
I20260808 08:24:06.766184 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:24:06.766515 12340 external_mini_cluster.cc:1428] Setting key 8a5a25d98debe40e4afd1c80cc19f48d
I20260808 08:24:06.769152 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36355
--builtin_ntp_servers=127.12.13.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 08:24:06.769345 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.760633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:43101 (local address 127.12.13.62:36355)
0808 08:24:06.760804 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:06.760811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:06.762008 (+ 1197us) server_negotiation.cc:407] Connection header received
0808 08:24:06.763560 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:06.763563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:06.763616 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:06.763700 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:06.764729 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.765289 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.767007 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.767364 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.767515 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.768755 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:06.768762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:06.768766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:06.768793 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:06.768821 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:06.768835 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:06.768926 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.769091 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.769161 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 08:24:06.770443 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.761053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36355 (local address 127.12.13.1:43101)
0808 08:24:06.761778 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:06.761832 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:06.763330 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:06.763887 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:06.763893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:06.763985 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:06.764564 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.764579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.765413 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.765415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.766859 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.766871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.767547 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.767550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.767725 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.768444 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:06.768446 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:06.768494 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:06.768571 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:06.768931 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:06.768939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:06.768944 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.769050 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.769446 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":127,"threads_started":1}
I20260808 08:24:06.771862 8335 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36355
I20260808 08:24:06.772166 8335 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:06.772913 8335 heartbeater.cc:511] Master 127.12.13.62:36355 requested a full tablet report, sending...
I20260808 08:24:06.774910 8051 ts_manager.cc:194] Registered new tserver with Master: 8a21097ff79b482fad71c4ec3160302c (127.12.13.1:43045)
I20260808 08:24:06.776191 8051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.1:43101
W20260808 08:24:06.893829 8339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:06.894181 8339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:06.894263 8339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:06.898519 8339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:06.898613 8339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:06.898707 8339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:24:06.904826 8339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:06.906165 8339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:06.907900 8339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:06.918330 8350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:06.918407 8347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:06.918570 8339 server_base.cc:1034] running on GCE node
W20260808 08:24:06.918406 8346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:06.919107 8339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:06.919893 8339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:06.921129 8339 hybrid_clock.cc:648] HybridClock initialized: now 1786177446921100 us; error 40 us; skew 500 ppm
I20260808 08:24:06.923489 8339 webserver.cc:460] Webserver started at http://127.12.13.2:35905/ using document root <none> and password file <none>
I20260808 08:24:06.924149 8339 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:06.924264 8339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:06.924520 8339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:06.926828 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "b8f54ddfdca442ffaf9401c0fd043ca5"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "d011109e0e0912e55a76c2ee99dc731d"
server_key_iv: "97e538513c0c79eaf48f6c7936fde100"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.927448 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "b8f54ddfdca442ffaf9401c0fd043ca5"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "d011109e0e0912e55a76c2ee99dc731d"
server_key_iv: "97e538513c0c79eaf48f6c7936fde100"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.931730 8339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:06.934608 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.936029 8339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:06.936210 8339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "b8f54ddfdca442ffaf9401c0fd043ca5"
format_stamp: "Formatted at 2026-08-08 08:24:06 on dist-test-slave-6ntk"
server_key: "d011109e0e0912e55a76c2ee99dc731d"
server_key_iv: "97e538513c0c79eaf48f6c7936fde100"
server_key_version: "encryptionkey@0"
I20260808 08:24:06.936416 8339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:06.967243 8339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:06.968146 8339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:06.968462 8339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:06.969270 8339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:06.970497 8339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:06.970587 8339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.970675 8339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:06.970728 8339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:06.981516 8339 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:42177
I20260808 08:24:06.981541 8514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:42177 every 8 connection(s)
I20260808 08:24:06.982671 8339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:24:06.989091 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 8339
I20260808 08:24:06.989264 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:24:06.989609 12340 external_mini_cluster.cc:1428] Setting key fa3b3ab4242338cf705ce8c4b3f65937
I20260808 08:24:06.991895 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36355
--builtin_ntp_servers=127.12.13.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 08:24:06.993818 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.984817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:44379 (local address 127.12.13.62:36355)
0808 08:24:06.984979 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:06.984984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:06.986194 (+ 1210us) server_negotiation.cc:407] Connection header received
0808 08:24:06.987790 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:06.987813 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:06.987897 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:06.987989 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:06.989030 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.989796 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.991469 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.991706 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.991833 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.993201 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:06.993204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:06.993207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:06.993222 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:06.993251 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:06.993260 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:06.993403 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:06.993589 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 08:24:06.993657 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:24:06.994853 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:06.985170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36355 (local address 127.12.13.2:44379)
0808 08:24:06.986029 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:06.986076 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:06.987518 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:06.988201 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:06.988209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:06.988308 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:06.988865 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.988878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.989957 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.989960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.991333 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:06.991344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:06.991875 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:06.991879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:06.992110 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:06.992882 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:06.992884 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:06.992958 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:06.993030 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:06.993404 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:06.993417 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:06.993426 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 08:24:06.993577 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 08:24:06.993914 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":144,"threads_started":1}
I20260808 08:24:06.996100 8515 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36355
I20260808 08:24:06.996371 8515 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:06.996965 8515 heartbeater.cc:511] Master 127.12.13.62:36355 requested a full tablet report, sending...
I20260808 08:24:06.998159 8051 ts_manager.cc:194] Registered new tserver with Master: b8f54ddfdca442ffaf9401c0fd043ca5 (127.12.13.2:42177)
I20260808 08:24:06.998772 8051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.2:44379
W20260808 08:24:07.125049 8522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:07.125351 8522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:07.125392 8522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:07.129647 8522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:07.129732 8522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:07.129799 8522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:24:07.135020 8522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:36355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:07.136329 8522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:07.137948 8522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:07.149150 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:07.149149 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:07.149199 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:07.149371 8522 server_base.cc:1034] running on GCE node
I20260808 08:24:07.150131 8522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:07.150859 8522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:07.152052 8522 hybrid_clock.cc:648] HybridClock initialized: now 1786177447152020 us; error 55 us; skew 500 ppm
I20260808 08:24:07.154209 8522 webserver.cc:460] Webserver started at http://127.12.13.3:41537/ using document root <none> and password file <none>
I20260808 08:24:07.154822 8522 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:07.154886 8522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:07.155071 8522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:07.157230 8522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "c477c3d2066e4399bc00bf4c130ab439"
format_stamp: "Formatted at 2026-08-08 08:24:07 on dist-test-slave-6ntk"
server_key: "4eef3bb63c85495449c3e2afe1444183"
server_key_iv: "295ce208eb7871e95ce99c9a1a304db2"
server_key_version: "encryptionkey@0"
I20260808 08:24:07.157778 8522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "c477c3d2066e4399bc00bf4c130ab439"
format_stamp: "Formatted at 2026-08-08 08:24:07 on dist-test-slave-6ntk"
server_key: "4eef3bb63c85495449c3e2afe1444183"
server_key_iv: "295ce208eb7871e95ce99c9a1a304db2"
server_key_version: "encryptionkey@0"
I20260808 08:24:07.161774 8522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 08:24:07.164578 8540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:07.166152 8522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 08:24:07.166306 8522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "c477c3d2066e4399bc00bf4c130ab439"
format_stamp: "Formatted at 2026-08-08 08:24:07 on dist-test-slave-6ntk"
server_key: "4eef3bb63c85495449c3e2afe1444183"
server_key_iv: "295ce208eb7871e95ce99c9a1a304db2"
server_key_version: "encryptionkey@0"
I20260808 08:24:07.166448 8522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:07.184525 8522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:07.185477 8522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:07.185703 8522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:07.186527 8522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:07.187642 8522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:07.187733 8522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:07.187814 8522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:07.187872 8522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:07.197929 8522 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:39407
I20260808 08:24:07.197961 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:39407 every 8 connection(s)
I20260808 08:24:07.198971 8522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:24:07.202193 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 8522
I20260808 08:24:07.202324 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:24:07.202626 12340 external_mini_cluster.cc:1428] Setting key 64c5119c16af637e63e9c885cb6e6ba9
I20260808 08:24:07.211158 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.201193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:54065 (local address 127.12.13.62:36355)
0808 08:24:07.201406 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.201410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.202925 (+ 1515us) server_negotiation.cc:407] Connection header received
0808 08:24:07.204874 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.204879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.204949 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.205058 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:07.206579 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.207201 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.208667 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.208936 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.209082 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.210568 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:07.210572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:07.210575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:07.210597 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:07.210621 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:07.210629 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:07.210720 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.210919 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.210994 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:24:07.212224 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.201667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:36355 (local address 127.12.13.3:54065)
0808 08:24:07.202698 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.202777 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:07.204610 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:07.205445 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:07.205453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:07.205604 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:07.206377 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.206396 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.207379 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.207384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.208477 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.208488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.209172 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.209180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.209395 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.210228 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:07.210230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:07.210280 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:07.210364 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:07.210722 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:07.210740 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:07.210750 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 08:24:07.210903 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 08:24:07.211245 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":213,"threads_started":1}
I20260808 08:24:07.213567 8699 heartbeater.cc:347] Connected to a master server at 127.12.13.62:36355
I20260808 08:24:07.213819 8699 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:07.214510 8699 heartbeater.cc:511] Master 127.12.13.62:36355 requested a full tablet report, sending...
I20260808 08:24:07.215845 8051 ts_manager.cc:194] Registered new tserver with Master: c477c3d2066e4399bc00bf4c130ab439 (127.12.13.3:39407)
I20260808 08:24:07.216610 8051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.3:54065
I20260808 08:24:07.217461 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:24:07.224200 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.219885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.12.13.62:36355)
0808 08:24:07.220046 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.220052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.220215 (+ 163us) server_negotiation.cc:407] Connection header received
0808 08:24:07.220330 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.220332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.220375 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.220446 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:07.220789 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.221423 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.222392 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.222600 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.222683 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.223754 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:07.223757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:07.223759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:07.223771 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:07.223789 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:07.223798 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:07.223909 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.224051 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.224084 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:24:07.227749 8057 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:24:07.231134 8057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:24:07.246729 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.241202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.12.13.2:42177)
0808 08:24:07.241944 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.241955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.241977 (+ 22us) server_negotiation.cc:407] Connection header received
0808 08:24:07.242050 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.242056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.242258 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.242407 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:07.242825 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.243948 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.245357 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.246030 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.246157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.246387 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.246446 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.246550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":181,"threads_started":1}
I20260808 08:24:07.246726 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.240524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:42177 (local address 127.0.0.1:44922)
0808 08:24:07.241226 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.241240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:07.241347 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:07.242567 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:07.242570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:07.242597 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:07.242693 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.242699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.244082 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.244085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.245226 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.245235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.246157 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.246161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.246276 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.246315 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 08:24:07.246421 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 08:24:07.246550 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":44,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20260808 08:24:07.246966 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.240790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:43045 (local address 127.0.0.1:54768)
0808 08:24:07.241668 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.241679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:07.241802 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:07.242529 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:07.242532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:07.242551 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:07.242677 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.242686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.243667 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.243673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.245066 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.245077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.246484 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.246487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.246574 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.246593 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 08:24:07.246666 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 08:24:07.246740 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":35,"spinlock_wait_cycles":1792,"thread_start_us":110,"threads_started":1}
I20260808 08:24:07.247126 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.240366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:39407 (local address 127.0.0.1:41702)
0808 08:24:07.241036 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.241087 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:07.241306 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:07.242199 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:07.242205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:07.242227 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:07.242394 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.242402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.244051 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.244054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.245259 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.245267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.246371 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.246375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.246470 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.246789 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 08:24:07.246859 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 08:24:07.246920 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":359,"thread_start_us":74,"threads_started":1}
I20260808 08:24:07.247313 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.241249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.12.13.1:43045)
0808 08:24:07.242052 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.242059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.242084 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:24:07.242171 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.242176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.242374 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.242552 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:07.242821 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.243549 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.245207 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.246102 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.246210 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.246429 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.247039 (+ 610us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.247141 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":94,"threads_started":1}
I20260808 08:24:07.247323 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.240512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.12.13.3:39407)
0808 08:24:07.241632 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.241645 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.241670 (+ 25us) server_negotiation.cc:407] Connection header received
0808 08:24:07.241756 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.241761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.242013 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.242560 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:07.242707 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.243912 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.245389 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.246231 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.246534 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.246724 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.247078 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.247163 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":114,"threads_started":1}
I20260808 08:24:07.250486 8246 tablet_service.cc:1467] Processing CreateTablet for tablet 06636b0e79f14417a7a2b6337deafb89 (DEFAULT_TABLE table=test-table [id=a7530cf3c9a743b893937bd53ea9bf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:07.250494 8605 tablet_service.cc:1467] Processing CreateTablet for tablet 06636b0e79f14417a7a2b6337deafb89 (DEFAULT_TABLE table=test-table [id=a7530cf3c9a743b893937bd53ea9bf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:07.251689 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06636b0e79f14417a7a2b6337deafb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:07.251690 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06636b0e79f14417a7a2b6337deafb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:07.250489 8423 tablet_service.cc:1467] Processing CreateTablet for tablet 06636b0e79f14417a7a2b6337deafb89 (DEFAULT_TABLE table=test-table [id=a7530cf3c9a743b893937bd53ea9bf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:07.252707 8423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06636b0e79f14417a7a2b6337deafb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:07.258445 8720 tablet_bootstrap.cc:492] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c: Bootstrap starting.
I20260808 08:24:07.260253 8722 tablet_bootstrap.cc:492] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Bootstrap starting.
I20260808 08:24:07.261040 8720 tablet_bootstrap.cc:654] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:07.261529 8723 tablet_bootstrap.cc:492] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Bootstrap starting.
I20260808 08:24:07.262010 8720 log.cc:824] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:07.262672 8722 tablet_bootstrap.cc:654] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:07.263530 8722 log.cc:824] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:07.264191 8723 tablet_bootstrap.cc:654] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:07.264784 8720 tablet_bootstrap.cc:492] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c: No bootstrap required, opened a new log
I20260808 08:24:07.265025 8720 ts_tablet_manager.cc:1397] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260808 08:24:07.265154 8723 log.cc:824] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:07.265825 8722 tablet_bootstrap.cc:492] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: No bootstrap required, opened a new log
I20260808 08:24:07.266078 8722 ts_tablet_manager.cc:1397] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260808 08:24:07.267500 8723 tablet_bootstrap.cc:492] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: No bootstrap required, opened a new log
I20260808 08:24:07.267731 8723 ts_tablet_manager.cc:1397] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:07.269434 8722 raft_consensus.cc:348] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.269703 8722 raft_consensus.cc:374] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:07.269753 8722 raft_consensus.cc:729] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c477c3d2066e4399bc00bf4c130ab439, State: Initialized, Role: FOLLOWER
I20260808 08:24:07.270315 8722 consensus_queue.cc:260] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.270716 8720 raft_consensus.cc:348] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.271178 8720 raft_consensus.cc:374] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:07.271247 8720 raft_consensus.cc:729] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a21097ff79b482fad71c4ec3160302c, State: Initialized, Role: FOLLOWER
I20260808 08:24:07.271553 8722 ts_tablet_manager.cc:1428] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260808 08:24:07.271967 8699 heartbeater.cc:503] Master 127.12.13.62:36355 was elected leader, sending a full tablet report...
I20260808 08:24:07.271868 8720 consensus_queue.cc:260] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.272672 8335 heartbeater.cc:503] Master 127.12.13.62:36355 was elected leader, sending a full tablet report...
I20260808 08:24:07.273028 8720 ts_tablet_manager.cc:1428] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20260808 08:24:07.273350 8723 raft_consensus.cc:348] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.273598 8723 raft_consensus.cc:374] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:07.273679 8723 raft_consensus.cc:729] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f54ddfdca442ffaf9401c0fd043ca5, State: Initialized, Role: FOLLOWER
I20260808 08:24:07.274163 8723 consensus_queue.cc:260] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.274984 8723 ts_tablet_manager.cc:1428] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260808 08:24:07.275408 8515 heartbeater.cc:503] Master 127.12.13.62:36355 was elected leader, sending a full tablet report...
I20260808 08:24:07.286075 8729 raft_consensus.cc:482] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:24:07.286303 8729 raft_consensus.cc:504] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.287496 8729 leader_election.cc:288] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c477c3d2066e4399bc00bf4c130ab439 (127.12.13.3:39407), b8f54ddfdca442ffaf9401c0fd043ca5 (127.12.13.2:42177)
I20260808 08:24:07.291846 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.287854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:43457 (local address 127.12.13.3:39407)
0808 08:24:07.287983 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.287988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.288266 (+ 278us) server_negotiation.cc:407] Connection header received
0808 08:24:07.288449 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.288452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.288504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.288598 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:07.288875 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.289390 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.290858 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.291251 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.291378 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.291441 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.291632 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.291701 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:24:07.291868 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.287761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:39407 (local address 127.12.13.1:43457)
0808 08:24:07.288160 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.288173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:07.288330 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:07.288612 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:07.288615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:07.288632 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:07.288764 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.288770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.289535 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.289538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.290721 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.290732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.291384 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.291388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.291483 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.291506 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:24:07.291603 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 08:24:07.291676 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"thread_start_us":150,"threads_started":1}
I20260808 08:24:07.291864 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.287854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:44919 (local address 127.12.13.2:42177)
0808 08:24:07.287983 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.287988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.288382 (+ 394us) server_negotiation.cc:407] Connection header received
0808 08:24:07.288543 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.288546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.288595 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.288676 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:07.288971 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.289464 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.290892 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.291278 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.291413 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.291473 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.291666 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.291742 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:24:07.291884 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.287761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:42177 (local address 127.12.13.1:44919)
0808 08:24:07.288239 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.288251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:07.288432 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:07.288699 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:07.288702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:07.288718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:07.288851 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.288857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.289619 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.289622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.290750 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:07.290762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.291418 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.291422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:07.291531 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.291554 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 08:24:07.291648 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 08:24:07.291718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":86,"thread_start_us":120,"threads_started":1}
I20260808 08:24:07.292538 8630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06636b0e79f14417a7a2b6337deafb89" candidate_uuid: "8a21097ff79b482fad71c4ec3160302c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c477c3d2066e4399bc00bf4c130ab439" is_pre_election: true
I20260808 08:24:07.292600 8449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06636b0e79f14417a7a2b6337deafb89" candidate_uuid: "8a21097ff79b482fad71c4ec3160302c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" is_pre_election: true
I20260808 08:24:07.292883 8630 raft_consensus.cc:2454] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a21097ff79b482fad71c4ec3160302c in term 0.
I20260808 08:24:07.292932 8449 raft_consensus.cc:2454] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a21097ff79b482fad71c4ec3160302c in term 0.
I20260808 08:24:07.293464 8173 leader_election.cc:302] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a21097ff79b482fad71c4ec3160302c, b8f54ddfdca442ffaf9401c0fd043ca5, c477c3d2066e4399bc00bf4c130ab439; no voters:
I20260808 08:24:07.293720 8729 raft_consensus.cc:2781] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:24:07.293824 8729 raft_consensus.cc:482] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:24:07.293905 8729 raft_consensus.cc:3037] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:07.294983 8729 raft_consensus.cc:504] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.295429 8729 leader_election.cc:288] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [CANDIDATE]: Term 1 election: Requested vote from peers c477c3d2066e4399bc00bf4c130ab439 (127.12.13.3:39407), b8f54ddfdca442ffaf9401c0fd043ca5 (127.12.13.2:42177)
I20260808 08:24:07.295717 8630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06636b0e79f14417a7a2b6337deafb89" candidate_uuid: "8a21097ff79b482fad71c4ec3160302c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c477c3d2066e4399bc00bf4c130ab439"
I20260808 08:24:07.295769 8449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06636b0e79f14417a7a2b6337deafb89" candidate_uuid: "8a21097ff79b482fad71c4ec3160302c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5"
I20260808 08:24:07.295849 8630 raft_consensus.cc:3037] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:07.295920 8449 raft_consensus.cc:3037] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:07.296991 8449 raft_consensus.cc:2454] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a21097ff79b482fad71c4ec3160302c in term 1.
I20260808 08:24:07.297012 8630 raft_consensus.cc:2454] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a21097ff79b482fad71c4ec3160302c in term 1.
I20260808 08:24:07.297329 8175 leader_election.cc:302] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a21097ff79b482fad71c4ec3160302c, b8f54ddfdca442ffaf9401c0fd043ca5; no voters:
I20260808 08:24:07.297514 8729 raft_consensus.cc:2781] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:07.297765 8729 raft_consensus.cc:686] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [term 1 LEADER]: Becoming Leader. State: Replica: 8a21097ff79b482fad71c4ec3160302c, State: Running, Role: LEADER
I20260808 08:24:07.298154 8729 consensus_queue.cc:237] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } }
I20260808 08:24:07.301819 8051 catalog_manager.cc:5495] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a21097ff79b482fad71c4ec3160302c (127.12.13.1). New cstate: current_term: 1 leader_uuid: "8a21097ff79b482fad71c4ec3160302c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a21097ff79b482fad71c4ec3160302c" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260808 08:24:07.329102 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.325230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.12.13.1:43045)
0808 08:24:07.325390 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:07.325395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:07.325412 (+ 17us) server_negotiation.cc:407] Connection header received
0808 08:24:07.325561 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:07.325564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:07.325626 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:07.325748 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:07.326096 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.326605 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.327518 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:07.327743 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:07.327863 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:07.328205 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:07.328638 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:07.328822 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:07.328909 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 08:24:07.328950 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:24:07.335451 8449 raft_consensus.cc:1261] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 1 FOLLOWER]: Refusing update from remote peer 8a21097ff79b482fad71c4ec3160302c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:07.335451 8630 raft_consensus.cc:1261] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 1 FOLLOWER]: Refusing update from remote peer 8a21097ff79b482fad71c4ec3160302c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:07.336297 8736 consensus_queue.cc:1035] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f54ddfdca442ffaf9401c0fd043ca5" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:07.336490 8729 consensus_queue.cc:1035] T 06636b0e79f14417a7a2b6337deafb89 P 8a21097ff79b482fad71c4ec3160302c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c477c3d2066e4399bc00bf4c130ab439" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:07.345203 8741 mvcc.cc:205] Tried to move back new op lower bound from 7316182824277954560 to 7316182824135966720. Current Snapshot: MvccSnapshot[applied={T|T < 7316182824277954560}]
I20260808 08:24:07.345502 8742 mvcc.cc:205] Tried to move back new op lower bound from 7316182824277954560 to 7316182824135966720. Current Snapshot: MvccSnapshot[applied={T|T < 7316182824277954560}]
I20260808 08:24:07.354562 8057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:07.354768 8057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:07.357429 8057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9604168c04a4c1f9b49c9fc9e23653b: Sending DeleteTablet for 3 replicas of tablet 06636b0e79f14417a7a2b6337deafb89
I20260808 08:24:07.358294 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet 06636b0e79f14417a7a2b6337deafb89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:07 UTC) from {username='slave'} at 127.0.0.1:41702
I20260808 08:24:07.358371 8423 tablet_service.cc:1514] Processing DeleteTablet for tablet 06636b0e79f14417a7a2b6337deafb89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:07 UTC) from {username='slave'} at 127.0.0.1:44922
I20260808 08:24:07.359061 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 8149
I20260808 08:24:07.370601 8754 tablet_replica.cc:330] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: stopping tablet replica
I20260808 08:24:07.370921 8753 tablet_replica.cc:330] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: stopping tablet replica
I20260808 08:24:07.371563 8753 raft_consensus.cc:2229] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:07.371680 8754 raft_consensus.cc:2229] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:07.372004 8753 raft_consensus.cc:2258] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:07.372028 8754 raft_consensus.cc:2258] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:07.373934 8753 ts_tablet_manager.cc:1905] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:07.375150 8754 ts_tablet_manager.cc:1905] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:07.377907 8753 ts_tablet_manager.cc:1918] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:07.378039 8753 log.cc:1196] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/06636b0e79f14417a7a2b6337deafb89
I20260808 08:24:07.378283 8754 ts_tablet_manager.cc:1918] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:07.378420 8754 log.cc:1196] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/06636b0e79f14417a7a2b6337deafb89
I20260808 08:24:07.378413 8753 ts_tablet_manager.cc:1939] T 06636b0e79f14417a7a2b6337deafb89 P b8f54ddfdca442ffaf9401c0fd043ca5: Deleting consensus metadata
I20260808 08:24:07.378712 8754 ts_tablet_manager.cc:1939] T 06636b0e79f14417a7a2b6337deafb89 P c477c3d2066e4399bc00bf4c130ab439: Deleting consensus metadata
I20260808 08:24:07.379951 8036 catalog_manager.cc:4841] TS c477c3d2066e4399bc00bf4c130ab439 (127.12.13.3:39407): tablet 06636b0e79f14417a7a2b6337deafb89 (table test-table [id=a7530cf3c9a743b893937bd53ea9bf44]) successfully deleted
I20260808 08:24:07.379951 8037 catalog_manager.cc:4841] TS b8f54ddfdca442ffaf9401c0fd043ca5 (127.12.13.2:42177): tablet 06636b0e79f14417a7a2b6337deafb89 (table test-table [id=a7530cf3c9a743b893937bd53ea9bf44]) successfully deleted
W20260808 08:24:07.380442 8034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:43045 (error 108)
I20260808 08:24:07.380466 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 8339
I20260808 08:24:07.391042 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:07.390663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:43045 (local address 127.0.0.1:54796)
0808 08:24:07.390836 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:07.390913 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 08:24:07.391263 8034 catalog_manager.cc:4584] TS 8a21097ff79b482fad71c4ec3160302c (127.12.13.1:43045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06636b0e79f14417a7a2b6337deafb89: Network error: Client connection negotiation failed: client connection to 127.12.13.1:43045: connect: Connection refused (error 111)
I20260808 08:24:07.395277 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 8522
I20260808 08:24:07.408324 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 8015
2026-08-08T08:24:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1182 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:07 dist-test-slave-6ntk krb5kdc[8758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:07 dist-test-slave-6ntk krb5kdc[8758](info): set up 2 sockets
Aug 08 08:24:07 dist-test-slave-6ntk krb5kdc[8758](info): commencing operation
krb5kdc: starting...
W20260808 08:24:08.670277 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:08 dist-test-slave-6ntk krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:08.814469 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:37971
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:08.948239 8779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:08.948621 8779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:08.948712 8779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:08.953284 8779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:08.953390 8779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:08.953431 8779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:08.953470 8779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:08.953506 8779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:08.959170 8779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:37971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:08.960398 8779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:08.961539 8779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:08.970937 8785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:08.970937 8784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:08.971045 8787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:08.971504 8779 server_base.cc:1034] running on GCE node
I20260808 08:24:08.972195 8779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:08.973491 8779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:08.974794 8779 hybrid_clock.cc:648] HybridClock initialized: now 1786177448974748 us; error 57 us; skew 500 ppm
Aug 08 08:24:08 dist-test-slave-6ntk krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177448, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:08.978344 8779 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:08.979383 8779 webserver.cc:460] Webserver started at http://127.12.13.62:36147/ using document root <none> and password file <none>
I20260808 08:24:08.979982 8779 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:08.980046 8779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:08.980235 8779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:08.982615 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "de25235ed8b34282aba480c741beb3fb"
format_stamp: "Formatted at 2026-08-08 08:24:08 on dist-test-slave-6ntk"
server_key: "53508bb0d02fcadf2c7e287fe07cff88"
server_key_iv: "93fbd2181a2182107bba91dff2ed6128"
server_key_version: "encryptionkey@0"
I20260808 08:24:08.983265 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "de25235ed8b34282aba480c741beb3fb"
format_stamp: "Formatted at 2026-08-08 08:24:08 on dist-test-slave-6ntk"
server_key: "53508bb0d02fcadf2c7e287fe07cff88"
server_key_iv: "93fbd2181a2182107bba91dff2ed6128"
server_key_version: "encryptionkey@0"
I20260808 08:24:08.987336 8779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:08.990072 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:08.991425 8779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:24:08.991580 8779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "de25235ed8b34282aba480c741beb3fb"
format_stamp: "Formatted at 2026-08-08 08:24:08 on dist-test-slave-6ntk"
server_key: "53508bb0d02fcadf2c7e287fe07cff88"
server_key_iv: "93fbd2181a2182107bba91dff2ed6128"
server_key_version: "encryptionkey@0"
I20260808 08:24:08.991699 8779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:09.003667 8779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:09.007285 8779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:09.007517 8779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:09.015575 8779 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:37971
I20260808 08:24:09.015640 8875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:37971 every 8 connection(s)
I20260808 08:24:09.016646 8779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:09.020059 8876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:09.023738 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 8779
I20260808 08:24:09.023933 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:09.024257 12340 external_mini_cluster.cc:1428] Setting key 797aa19afa05e0f506540255ca56d5a2
I20260808 08:24:09.027365 8876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: Bootstrap starting.
I20260808 08:24:09.030571 8876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:09.031548 8876 log.cc:824] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:09.033980 8876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: No bootstrap required, opened a new log
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:09.038967 8876 raft_consensus.cc:348] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de25235ed8b34282aba480c741beb3fb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 37971 } }
I20260808 08:24:09.039328 8876 raft_consensus.cc:374] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:09.039388 8876 raft_consensus.cc:729] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de25235ed8b34282aba480c741beb3fb, State: Initialized, Role: FOLLOWER
I20260808 08:24:09.039992 8876 consensus_queue.cc:260] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de25235ed8b34282aba480c741beb3fb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 37971 } }
I20260808 08:24:09.040256 8876 raft_consensus.cc:388] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:09.040352 8876 raft_consensus.cc:482] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:09.040490 8876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:09.041726 8876 raft_consensus.cc:504] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de25235ed8b34282aba480c741beb3fb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 37971 } }
I20260808 08:24:09.042192 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.026034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.12.13.62:37971)
0808 08:24:09.026597 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.026618 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.026679 (+ 61us) server_negotiation.cc:407] Connection header received
0808 08:24:09.027617 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.027648 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.028135 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.028547 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:09.029270 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.031008 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.032012 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.032696 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.032869 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.035562 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:09.035569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:09.035599 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:09.035634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:09.038732 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.039217 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.039224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.039231 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.039406 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.039744 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.039748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.039750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.040287 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:09.040544 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.040898 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.041241 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":174,"threads_started":1}
I20260808 08:24:09.042176 8876 leader_election.cc:302] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de25235ed8b34282aba480c741beb3fb; no voters:
I20260808 08:24:09.042706 8876 leader_election.cc:288] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:09.042807 8882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:09.043041 8882 raft_consensus.cc:686] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [term 1 LEADER]: Becoming Leader. State: Replica: de25235ed8b34282aba480c741beb3fb, State: Running, Role: LEADER
I20260808 08:24:09.044332 8876 sys_catalog.cc:564] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:09.043401 8882 consensus_queue.cc:237] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de25235ed8b34282aba480c741beb3fb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 37971 } }
I20260808 08:24:09.046640 8883 sys_catalog.cc:455] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader de25235ed8b34282aba480c741beb3fb. Latest consensus state: current_term: 1 leader_uuid: "de25235ed8b34282aba480c741beb3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de25235ed8b34282aba480c741beb3fb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 37971 } } }
I20260808 08:24:09.046841 8883 sys_catalog.cc:458] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:09.047446 8884 sys_catalog.cc:455] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de25235ed8b34282aba480c741beb3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de25235ed8b34282aba480c741beb3fb" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 37971 } } }
I20260808 08:24:09.047546 8884 sys_catalog.cc:458] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:09.049959 8899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:09.050058 8899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:09.050148 8900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:09.050956 8900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:09.056958 8900 catalog_manager.cc:1347] Generated new cluster ID: 3322777a372c44ae88e35c4b0e61a1b0
I20260808 08:24:09.057044 8900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:09.070981 8900 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:09.072438 8900 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:09.081919 8900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: Generated new TSK 0
I20260808 08:24:09.082902 8900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
Entry for principal kudu/127.12.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab.
I20260808 08:24:09.122241 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:37971
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:09.251739 8912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:09.252048 8912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:09.252095 8912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:09.256347 8912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:09.256425 8912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:09.256487 8912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:24:09.262462 8912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.1.keytab
--principal=kudu/127.12.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:09.263717 8912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:09.264890 8912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:09.275967 8923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:09.275959 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:09.275959 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:09.276072 8912 server_base.cc:1034] running on GCE node
I20260808 08:24:09.276763 8912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:09.277534 8912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:09.278751 8912 hybrid_clock.cc:648] HybridClock initialized: now 1786177449278707 us; error 57 us; skew 500 ppm
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177449, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:09.282392 8912 init.cc:377] Logged in from keytab as kudu/127.12.13.1@KRBTEST.COM (short username kudu)
I20260808 08:24:09.283466 8912 webserver.cc:460] Webserver started at http://127.12.13.1:34625/ using document root <none> and password file <none>
I20260808 08:24:09.284157 8912 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:09.284265 8912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:09.284528 8912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:09.286921 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "51f04a0d8cdb477c9aa882b564ad17c7"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "027bbf22b733e38e3855d39f2b2cf30e"
server_key_iv: "bdaf8e4d8b604776dfffa8148587af15"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.287545 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "51f04a0d8cdb477c9aa882b564ad17c7"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "027bbf22b733e38e3855d39f2b2cf30e"
server_key_iv: "bdaf8e4d8b604776dfffa8148587af15"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.291666 8912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 08:24:09.294544 8933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.296003 8912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:09.296159 8912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "51f04a0d8cdb477c9aa882b564ad17c7"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "027bbf22b733e38e3855d39f2b2cf30e"
server_key_iv: "bdaf8e4d8b604776dfffa8148587af15"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.296293 8912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:09.316424 8912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:09.319967 8912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:09.320191 8912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:09.320884 8912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:09.321798 8912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:09.321877 8912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.322000 8912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:09.322057 8912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.332490 8912 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:41259
I20260808 08:24:09.332513 9087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:41259 every 8 connection(s)
I20260808 08:24:09.333554 8912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:24:09.341516 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 8912
I20260808 08:24:09.341711 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:24:09.342165 12340 external_mini_cluster.cc:1428] Setting key 285195089d19c9a4127ff9b50106d924
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177449, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.1@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:09.349181 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.335567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:34829 (local address 127.12.13.62:37971)
0808 08:24:09.335752 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.335757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.336644 (+ 887us) server_negotiation.cc:407] Connection header received
0808 08:24:09.338325 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.338328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.338383 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.338484 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:09.339894 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.340561 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.341925 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.342321 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.342447 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.345298 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:09.345302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:09.345304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:09.345336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:09.346995 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.347577 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.347582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.347584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.347668 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.348035 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.348040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.348041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.348234 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:09.348354 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.348881 (+ 527us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.349014 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 08:24:09.350265 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.335920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:37971 (local address 127.12.13.1:34829)
0808 08:24:09.336467 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.336514 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.338057 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.338694 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.338702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.339112 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:09.339726 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.339742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.340712 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.340714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.341652 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.341711 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.342483 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.342486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.342630 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.343269 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:09.343291 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:09.345115 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:09.347129 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:09.347136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:09.347152 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:09.347463 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:09.347767 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:09.347771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:09.347773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:09.347931 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:09.348353 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:09.348362 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:09.348628 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.348833 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.349182 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260808 08:24:09.351744 9088 heartbeater.cc:347] Connected to a master server at 127.12.13.62:37971
I20260808 08:24:09.352022 9088 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:09.352559 9088 heartbeater.cc:511] Master 127.12.13.62:37971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.13.2@KRBTEST.COM; defaulting to no policy
I20260808 08:24:09.354319 8831 ts_manager.cc:194] Registered new tserver with Master: 51f04a0d8cdb477c9aa882b564ad17c7 (127.12.13.1:41259)
I20260808 08:24:09.355594 8831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.1@KRBTEST.COM'} at 127.12.13.1:34829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
Entry for principal kudu/127.12.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab.
I20260808 08:24:09.377985 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:37971
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:09.507972 9093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:09.508340 9093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:09.508425 9093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:09.513379 9093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:09.513487 9093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:09.513595 9093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:24:09.519418 9093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.2.keytab
--principal=kudu/127.12.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:09.520694 9093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:09.521822 9093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:09.532158 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:09.532214 9102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:09.532275 9093 server_base.cc:1034] running on GCE node
W20260808 08:24:09.532183 9101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:09.532954 9093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:09.533644 9093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:09.534837 9093 hybrid_clock.cc:648] HybridClock initialized: now 1786177449534811 us; error 45 us; skew 500 ppm
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177449, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:09.538010 9093 init.cc:377] Logged in from keytab as kudu/127.12.13.2@KRBTEST.COM (short username kudu)
I20260808 08:24:09.539012 9093 webserver.cc:460] Webserver started at http://127.12.13.2:43231/ using document root <none> and password file <none>
I20260808 08:24:09.539640 9093 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:09.539706 9093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:09.539951 9093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:09.542398 9093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "4a17a62b0ef741ae9c7ca727420f46f2"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "902d2cfc63a7e83883795581c65958aa"
server_key_iv: "0fff0b6fbd7715d23cc2603835611a44"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.543020 9093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "4a17a62b0ef741ae9c7ca727420f46f2"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "902d2cfc63a7e83883795581c65958aa"
server_key_iv: "0fff0b6fbd7715d23cc2603835611a44"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.547806 9093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:09.550654 9115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.552040 9093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:09.552205 9093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "4a17a62b0ef741ae9c7ca727420f46f2"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "902d2cfc63a7e83883795581c65958aa"
server_key_iv: "0fff0b6fbd7715d23cc2603835611a44"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.552346 9093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:09.571631 9093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:09.575059 9093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:09.575347 9093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:09.576040 9093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:09.576995 9093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:09.577044 9093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.577088 9093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:09.577104 9093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.587199 9093 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:44205
I20260808 08:24:09.587255 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:44205 every 8 connection(s)
I20260808 08:24:09.588174 9093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:24:09.596704 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 9093
I20260808 08:24:09.596848 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:24:09.597131 12340 external_mini_cluster.cc:1428] Setting key ba0706d6498dc212a9537fabec737280
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177449, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.2@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:09.603233 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.590174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:50427 (local address 127.12.13.62:37971)
0808 08:24:09.590347 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.590352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.591270 (+ 918us) server_negotiation.cc:407] Connection header received
0808 08:24:09.592628 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.592630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.592677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.592754 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:09.594043 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.594584 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.595786 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.596053 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.596145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.599052 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:09.599056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:09.599057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:09.599082 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:09.600685 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.601375 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.601380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.601384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.601486 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.601921 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.601927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.601931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.602193 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:09.602320 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.602834 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.602979 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:24:09.603967 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.590616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:37971 (local address 127.12.13.2:50427)
0808 08:24:09.591101 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.591149 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.592421 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.592924 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.592931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.593334 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:09.593835 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.593849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.594705 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.594708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.595680 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.595689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.596202 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.596205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.596385 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.597103 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:09.597121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:09.598876 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:09.600806 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:09.600815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:09.600835 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:09.601228 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:09.601587 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:09.601590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:09.601591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:09.601771 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:09.602313 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:09.602320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:09.602575 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.602740 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.603013 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260808 08:24:09.605254 9267 heartbeater.cc:347] Connected to a master server at 127.12.13.62:37971
I20260808 08:24:09.605549 9267 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:09.606320 9267 heartbeater.cc:511] Master 127.12.13.62:37971 requested a full tablet report, sending...
I20260808 08:24:09.607643 8831 ts_manager.cc:194] Registered new tserver with Master: 4a17a62b0ef741ae9c7ca727420f46f2 (127.12.13.2:44205)
I20260808 08:24:09.608274 8831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.2@KRBTEST.COM'} at 127.12.13.2:50427
WARNING: no policy specified for kudu/127.12.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
Entry for principal kudu/127.12.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab.
I20260808 08:24:09.634591 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:37971
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:09.755203 9275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:09.755505 9275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:09.755594 9275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:09.759773 9275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:09.759843 9275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:09.759904 9275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:24:09.765497 9275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.3.keytab
--principal=kudu/127.12.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:09.766788 9275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:09.767931 9275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:09.778700 9284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:09.778710 9281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:09.778679 9282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:09.779073 9275 server_base.cc:1034] running on GCE node
I20260808 08:24:09.779605 9275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:09.780397 9275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:09.781636 9275 hybrid_clock.cc:648] HybridClock initialized: now 1786177449781618 us; error 30 us; skew 500 ppm
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177449, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:09.784966 9275 init.cc:377] Logged in from keytab as kudu/127.12.13.3@KRBTEST.COM (short username kudu)
I20260808 08:24:09.786248 9275 webserver.cc:460] Webserver started at http://127.12.13.3:35121/ using document root <none> and password file <none>
I20260808 08:24:09.786935 9275 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:09.787062 9275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:09.787334 9275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:09.789577 9275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "83c48e9f1ddb4e65ad69605a306d29f2"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "3a818fb579b420a9d0a292046ee90ce9"
server_key_iv: "e9623199868a31628253f5e520bade98"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.790228 9275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "83c48e9f1ddb4e65ad69605a306d29f2"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "3a818fb579b420a9d0a292046ee90ce9"
server_key_iv: "e9623199868a31628253f5e520bade98"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.794435 9275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 08:24:09.797221 9295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.798599 9275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:09.798758 9275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "83c48e9f1ddb4e65ad69605a306d29f2"
format_stamp: "Formatted at 2026-08-08 08:24:09 on dist-test-slave-6ntk"
server_key: "3a818fb579b420a9d0a292046ee90ce9"
server_key_iv: "e9623199868a31628253f5e520bade98"
server_key_version: "encryptionkey@0"
I20260808 08:24:09.798888 9275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:09.826344 9275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:09.829813 9275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:09.830084 9275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:09.830837 9275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:09.831795 9275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:09.831874 9275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.831948 9275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:09.831984 9275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:09.842490 9275 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:35465
I20260808 08:24:09.842514 9453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:35465 every 8 connection(s)
I20260808 08:24:09.843456 9275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:24:09.853585 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 9275
I20260808 08:24:09.853750 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:24:09.854118 12340 external_mini_cluster.cc:1428] Setting key 10aba59f539e0a83fa88b82e44c326c3
Aug 08 08:24:09 dist-test-slave-6ntk krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177449, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.3@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:09.859422 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.845622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:60397 (local address 127.12.13.62:37971)
0808 08:24:09.845785 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.845789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.846896 (+ 1107us) server_negotiation.cc:407] Connection header received
0808 08:24:09.848333 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.848337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.848399 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.848520 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:09.849769 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.850391 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.851719 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.851947 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.852067 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.855167 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:09.855171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:09.855174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:09.855207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:09.857200 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.857742 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.857747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.857749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.857853 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.858250 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.858255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.858257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.858459 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:09.858572 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.859123 (+ 551us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.859254 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 08:24:09.860366 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.846188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:37971 (local address 127.12.13.3:60397)
0808 08:24:09.846704 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.846751 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.848091 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.848704 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.848712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.849105 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:09.849604 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.849619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.850540 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.850548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.851584 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.851594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.852113 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.852117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.852291 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.852951 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:09.852968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:09.854913 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:09.857322 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:09.857329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:09.857347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:09.857631 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:09.857981 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:09.857985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:09.857987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:09.858147 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:09.858572 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:09.858581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:09.858875 (+ 294us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.859081 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.859445 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":124,"threads_started":1}
I20260808 08:24:09.861649 9454 heartbeater.cc:347] Connected to a master server at 127.12.13.62:37971
I20260808 08:24:09.861982 9454 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:09.862483 9454 heartbeater.cc:511] Master 127.12.13.62:37971 requested a full tablet report, sending...
I20260808 08:24:09.863572 8831 ts_manager.cc:194] Registered new tserver with Master: 83c48e9f1ddb4e65ad69605a306d29f2 (127.12.13.3:35465)
I20260808 08:24:09.864113 8831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.13.3@KRBTEST.COM'} at 127.12.13.3:60397
I20260808 08:24:09.868403 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:24:09.878875 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.870597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.12.13.62:37971)
0808 08:24:09.870810 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.870816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.870978 (+ 162us) server_negotiation.cc:407] Connection header received
0808 08:24:09.871145 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.871149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.871261 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.871369 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:09.872167 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.872706 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.873597 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.873823 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.873985 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.875487 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:09.875493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:09.875501 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:09.875526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:09.877005 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.877457 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.877461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.877463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.877551 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:09.877863 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:09.877886 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:09.877891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:09.878139 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:09.878336 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.878570 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.878693 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 08:24:09.882431 8831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:24:09.885691 8831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:24:09.901284 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.895878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:35465 (local address 127.0.0.1:47296)
0808 08:24:09.896816 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.896831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.896929 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.897471 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.897474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.897493 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:09.897589 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.897594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.898705 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.898711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.899757 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.899766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.900857 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.900860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.900948 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.900972 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.901051 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.901121 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":107,"threads_started":1}
I20260808 08:24:09.901393 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.896135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.12.13.3:35465)
0808 08:24:09.896662 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.896675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.897028 (+ 353us) server_negotiation.cc:407] Connection header received
0808 08:24:09.897113 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.897118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.897318 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.897478 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:09.897687 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.898567 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.899896 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.900676 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.900796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.900961 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.901101 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.901208 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":200,"threads_started":1}
I20260808 08:24:09.901675 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.895267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.12.13.1:41259)
0808 08:24:09.895585 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.895595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.895629 (+ 34us) server_negotiation.cc:407] Connection header received
0808 08:24:09.895901 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.895907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.896128 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.896305 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:09.896864 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.897851 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.899551 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.900427 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.900547 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.900762 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.901201 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.901298 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260808 08:24:09.901753 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.895062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:41259 (local address 127.0.0.1:50776)
0808 08:24:09.895498 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.895558 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.895709 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.896441 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.896447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.896470 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:09.896629 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.896637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.898006 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.898012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.899419 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.899429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.900553 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.900557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.900675 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.900766 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.900911 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.901085 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260808 08:24:09.902437 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.896227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:44205 (local address 127.0.0.1:47824)
0808 08:24:09.896950 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.896964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.897051 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.897818 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.897821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.897841 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:09.897948 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.897954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.899374 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.899377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.900497 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.900509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.902045 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.902048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.902139 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.902161 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.902236 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.902306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":88,"threads_started":1}
I20260808 08:24:09.903033 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.896666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.12.13.2:44205)
0808 08:24:09.897148 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.897160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.897179 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:24:09.897245 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.897249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.897408 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.897644 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:09.898057 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.899208 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.900634 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.901751 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.901887 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.902045 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.902640 (+ 595us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.902755 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":140,"threads_started":1}
I20260808 08:24:09.904812 9357 tablet_service.cc:1467] Processing CreateTablet for tablet 888622e953b44e2e8a3c4fffd5334b27 (DEFAULT_TABLE table=test-table [id=2cd6f81ebcc14e3480b33e4759206721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:09.905084 8999 tablet_service.cc:1467] Processing CreateTablet for tablet 888622e953b44e2e8a3c4fffd5334b27 (DEFAULT_TABLE table=test-table [id=2cd6f81ebcc14e3480b33e4759206721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:09.906044 9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888622e953b44e2e8a3c4fffd5334b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:09.905709 9178 tablet_service.cc:1467] Processing CreateTablet for tablet 888622e953b44e2e8a3c4fffd5334b27 (DEFAULT_TABLE table=test-table [id=2cd6f81ebcc14e3480b33e4759206721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:09.906224 8999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888622e953b44e2e8a3c4fffd5334b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:09.906752 9178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888622e953b44e2e8a3c4fffd5334b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:09.912837 9475 tablet_bootstrap.cc:492] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Bootstrap starting.
I20260808 08:24:09.914033 9476 tablet_bootstrap.cc:492] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7: Bootstrap starting.
I20260808 08:24:09.915500 9475 tablet_bootstrap.cc:654] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:09.915830 9476 tablet_bootstrap.cc:654] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:09.916563 9476 log.cc:824] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:09.916710 9475 log.cc:824] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:09.917390 9477 tablet_bootstrap.cc:492] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Bootstrap starting.
I20260808 08:24:09.918725 9475 tablet_bootstrap.cc:492] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: No bootstrap required, opened a new log
I20260808 08:24:09.918856 9476 tablet_bootstrap.cc:492] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7: No bootstrap required, opened a new log
I20260808 08:24:09.918989 9475 ts_tablet_manager.cc:1397] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:09.919046 9476 ts_tablet_manager.cc:1397] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 08:24:09.919626 9477 tablet_bootstrap.cc:654] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:09.921034 9477 log.cc:824] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:09.922353 9475 raft_consensus.cc:348] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.922617 9475 raft_consensus.cc:374] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:09.922674 9475 raft_consensus.cc:729] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c48e9f1ddb4e65ad69605a306d29f2, State: Initialized, Role: FOLLOWER
I20260808 08:24:09.923143 9475 consensus_queue.cc:260] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.923293 9477 tablet_bootstrap.cc:492] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: No bootstrap required, opened a new log
I20260808 08:24:09.923499 9477 ts_tablet_manager.cc:1397] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:09.923702 9476 raft_consensus.cc:348] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.923959 9476 raft_consensus.cc:374] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:09.924011 9476 raft_consensus.cc:729] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51f04a0d8cdb477c9aa882b564ad17c7, State: Initialized, Role: FOLLOWER
I20260808 08:24:09.924249 9475 ts_tablet_manager.cc:1428] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 08:24:09.924666 9454 heartbeater.cc:503] Master 127.12.13.62:37971 was elected leader, sending a full tablet report...
I20260808 08:24:09.924485 9476 consensus_queue.cc:260] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.926419 9088 heartbeater.cc:503] Master 127.12.13.62:37971 was elected leader, sending a full tablet report...
I20260808 08:24:09.926812 9476 ts_tablet_manager.cc:1428] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 08:24:09.927596 9477 raft_consensus.cc:348] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.927915 9477 raft_consensus.cc:374] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:09.927980 9477 raft_consensus.cc:729] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17a62b0ef741ae9c7ca727420f46f2, State: Initialized, Role: FOLLOWER
I20260808 08:24:09.928480 9477 consensus_queue.cc:260] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.929252 9267 heartbeater.cc:503] Master 127.12.13.62:37971 was elected leader, sending a full tablet report...
I20260808 08:24:09.929569 9477 ts_tablet_manager.cc:1428] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:24:09.940448 9481 raft_consensus.cc:482] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:24:09.940699 9481 raft_consensus.cc:504] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.941993 9481 leader_election.cc:288] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51f04a0d8cdb477c9aa882b564ad17c7 (127.12.13.1:41259), 4a17a62b0ef741ae9c7ca727420f46f2 (127.12.13.2:44205)
I20260808 08:24:09.947031 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.942429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:34017 (local address 127.12.13.1:41259)
0808 08:24:09.942579 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.942586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.942602 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:24:09.942661 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.942664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.942713 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.942784 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:09.943197 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.944017 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.945596 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.946327 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.946459 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.946529 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.946812 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.946893 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 08:24:09.947031 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.942675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:36001 (local address 127.12.13.2:44205)
0808 08:24:09.942813 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.942818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.943035 (+ 217us) server_negotiation.cc:407] Connection header received
0808 08:24:09.943125 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.943128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.943180 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.943261 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:09.943625 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.944287 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.945660 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.946341 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.946458 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.946529 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.946824 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.946893 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:24:09.947045 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.942227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:41259 (local address 127.12.13.3:34017)
0808 08:24:09.942382 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.942399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.942507 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.942893 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.942896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.942920 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:09.943077 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.943087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.944152 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.944155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.945474 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.945486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.946462 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.946466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.946635 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.946681 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.946768 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.946864 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":10}
I20260808 08:24:09.947058 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.942516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:44205 (local address 127.12.13.3:36001)
0808 08:24:09.942931 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:09.942945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:09.943043 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:09.943331 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:09.943334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:09.943354 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:09.943490 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.943496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.944388 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.944391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.945546 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:09.945557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.946462 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.946466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:09.946650 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.946700 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 08:24:09.946768 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 08:24:09.946868 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":102,"thread_start_us":135,"threads_started":1}
I20260808 08:24:09.947786 9024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888622e953b44e2e8a3c4fffd5334b27" candidate_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" is_pre_election: true
I20260808 08:24:09.947971 9204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888622e953b44e2e8a3c4fffd5334b27" candidate_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" is_pre_election: true
I20260808 08:24:09.948170 9024 raft_consensus.cc:2454] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c48e9f1ddb4e65ad69605a306d29f2 in term 0.
I20260808 08:24:09.948235 9204 raft_consensus.cc:2454] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c48e9f1ddb4e65ad69605a306d29f2 in term 0.
I20260808 08:24:09.948720 9300 leader_election.cc:302] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51f04a0d8cdb477c9aa882b564ad17c7, 83c48e9f1ddb4e65ad69605a306d29f2; no voters:
I20260808 08:24:09.949059 9481 raft_consensus.cc:2781] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:24:09.949144 9481 raft_consensus.cc:482] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:24:09.949175 9481 raft_consensus.cc:3037] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:09.950350 9481 raft_consensus.cc:504] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.950708 9481 leader_election.cc:288] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [CANDIDATE]: Term 1 election: Requested vote from peers 51f04a0d8cdb477c9aa882b564ad17c7 (127.12.13.1:41259), 4a17a62b0ef741ae9c7ca727420f46f2 (127.12.13.2:44205)
I20260808 08:24:09.951102 9024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888622e953b44e2e8a3c4fffd5334b27" candidate_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51f04a0d8cdb477c9aa882b564ad17c7"
I20260808 08:24:09.951228 9024 raft_consensus.cc:3037] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:09.951187 9204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888622e953b44e2e8a3c4fffd5334b27" candidate_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17a62b0ef741ae9c7ca727420f46f2"
I20260808 08:24:09.951400 9204 raft_consensus.cc:3037] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:09.952344 9024 raft_consensus.cc:2454] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c48e9f1ddb4e65ad69605a306d29f2 in term 1.
I20260808 08:24:09.952544 9204 raft_consensus.cc:2454] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c48e9f1ddb4e65ad69605a306d29f2 in term 1.
I20260808 08:24:09.952706 9300 leader_election.cc:302] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51f04a0d8cdb477c9aa882b564ad17c7, 83c48e9f1ddb4e65ad69605a306d29f2; no voters:
I20260808 08:24:09.952890 9481 raft_consensus.cc:2781] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:09.953122 9481 raft_consensus.cc:686] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c48e9f1ddb4e65ad69605a306d29f2, State: Running, Role: LEADER
I20260808 08:24:09.953487 9481 consensus_queue.cc:237] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } }
I20260808 08:24:09.957113 8831 catalog_manager.cc:5495] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c48e9f1ddb4e65ad69605a306d29f2 (127.12.13.3). New cstate: current_term: 1 leader_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c48e9f1ddb4e65ad69605a306d29f2" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 } health_report { overall_health: UNKNOWN } } }
I20260808 08:24:09.981212 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:09.977575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.12.13.3:35465)
0808 08:24:09.977716 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:09.977720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:09.977734 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:24:09.977816 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:09.977818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:09.977941 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:09.978041 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:09.978325 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.978927 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.979780 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:09.980016 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:09.980105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:09.980382 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:09.980851 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:09.980948 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:09.981015 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 08:24:09.981070 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:24:09.988024 9024 raft_consensus.cc:1261] T 888622e953b44e2e8a3c4fffd5334b27 P 51f04a0d8cdb477c9aa882b564ad17c7 [term 1 FOLLOWER]: Refusing update from remote peer 83c48e9f1ddb4e65ad69605a306d29f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:09.988024 9204 raft_consensus.cc:1261] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 1 FOLLOWER]: Refusing update from remote peer 83c48e9f1ddb4e65ad69605a306d29f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:09.988870 9481 consensus_queue.cc:1035] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f04a0d8cdb477c9aa882b564ad17c7" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:09.989084 9487 consensus_queue.cc:1035] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17a62b0ef741ae9c7ca727420f46f2" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:10.005839 8831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:10.006102 8831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:10.008965 8831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de25235ed8b34282aba480c741beb3fb: Sending DeleteTablet for 3 replicas of tablet 888622e953b44e2e8a3c4fffd5334b27
I20260808 08:24:10.010159 9357 tablet_service.cc:1514] Processing DeleteTablet for tablet 888622e953b44e2e8a3c4fffd5334b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:10 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:47296
I20260808 08:24:10.010284 9178 tablet_service.cc:1514] Processing DeleteTablet for tablet 888622e953b44e2e8a3c4fffd5334b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:10 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:47824
I20260808 08:24:10.010789 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 8912
I20260808 08:24:10.010818 8999 tablet_service.cc:1514] Processing DeleteTablet for tablet 888622e953b44e2e8a3c4fffd5334b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:10 UTC) from {username='kudu', principal='kudu/127.12.13.62@KRBTEST.COM'} at 127.0.0.1:50776
I20260808 08:24:10.010896 9505 tablet_replica.cc:330] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: stopping tablet replica
I20260808 08:24:10.011377 9505 raft_consensus.cc:2229] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:10.011584 9503 tablet_replica.cc:330] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: stopping tablet replica
I20260808 08:24:10.011798 9505 raft_consensus.cc:2258] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:10.012014 9503 raft_consensus.cc:2229] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:24:10.012423 9503 raft_consensus.cc:2258] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:10.015455 9505 ts_tablet_manager.cc:1905] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:10.019605 9505 ts_tablet_manager.cc:1918] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:10.019723 9505 log.cc:1196] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/888622e953b44e2e8a3c4fffd5334b27
I20260808 08:24:10.020107 9505 ts_tablet_manager.cc:1939] T 888622e953b44e2e8a3c4fffd5334b27 P 4a17a62b0ef741ae9c7ca727420f46f2: Deleting consensus metadata
I20260808 08:24:10.021703 8802 catalog_manager.cc:4841] TS 4a17a62b0ef741ae9c7ca727420f46f2 (127.12.13.2:44205): tablet 888622e953b44e2e8a3c4fffd5334b27 (table test-table [id=2cd6f81ebcc14e3480b33e4759206721]) successfully deleted
I20260808 08:24:10.022210 9503 ts_tablet_manager.cc:1905] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:10.026225 9503 ts_tablet_manager.cc:1918] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:10.026387 9503 log.cc:1196] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/888622e953b44e2e8a3c4fffd5334b27
I20260808 08:24:10.026822 9503 ts_tablet_manager.cc:1939] T 888622e953b44e2e8a3c4fffd5334b27 P 83c48e9f1ddb4e65ad69605a306d29f2: Deleting consensus metadata
I20260808 08:24:10.028427 8802 catalog_manager.cc:4841] TS 83c48e9f1ddb4e65ad69605a306d29f2 (127.12.13.3:35465): tablet 888622e953b44e2e8a3c4fffd5334b27 (table test-table [id=2cd6f81ebcc14e3480b33e4759206721]) successfully deleted
W20260808 08:24:10.029412 8802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:41259 (error 108)
I20260808 08:24:10.030465 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 9093
I20260808 08:24:10.030654 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:10.030344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:41259 (local address 127.0.0.1:50788)
0808 08:24:10.030484 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:10.030567 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:41259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 08:24:10.033490 8802 catalog_manager.cc:4584] TS 51f04a0d8cdb477c9aa882b564ad17c7 (127.12.13.1:41259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 888622e953b44e2e8a3c4fffd5334b27: Network error: Client connection negotiation failed: client connection to 127.12.13.1:41259: connect: Connection refused (error 111)
I20260808 08:24:10.041481 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 9275
I20260808 08:24:10.062703 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 8779
2026-08-08T08:24:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2656 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T08:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:10Z Disabled control of system clock
I20260808 08:24:10.104846 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33735
--webserver_interface=127.12.13.62
--webserver_port=0
--builtin_ntp_servers=127.12.13.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 08:24:10.241297 9514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:10.241608 9514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:10.241647 9514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:10.246618 9514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:10.246696 9514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:10.246716 9514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:10.246737 9514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:10.246754 9514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:10.252079 9514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:10.253263 9514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:10.254769 9514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:10.264287 9525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:10.264334 9521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:10.264381 9514 server_base.cc:1034] running on GCE node
W20260808 08:24:10.264329 9520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:10.265250 9514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:10.266624 9514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:10.267836 9514 hybrid_clock.cc:648] HybridClock initialized: now 1786177450267807 us; error 43 us; skew 500 ppm
I20260808 08:24:10.270166 9514 webserver.cc:460] Webserver started at http://127.12.13.62:40405/ using document root <none> and password file <none>
I20260808 08:24:10.270880 9514 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:10.271106 9514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:10.271425 9514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:10.273699 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "298fa819555c4010b20ee44b22e595a4"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "0099ad691f5b8ce2b9eb48f2557287be"
server_key_iv: "ac1ce190eddb9d41d9576c1a98e377dd"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.274343 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "298fa819555c4010b20ee44b22e595a4"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "0099ad691f5b8ce2b9eb48f2557287be"
server_key_iv: "ac1ce190eddb9d41d9576c1a98e377dd"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.279059 9514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:10.282039 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.283841 9514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:24:10.283983 9514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "298fa819555c4010b20ee44b22e595a4"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "0099ad691f5b8ce2b9eb48f2557287be"
server_key_iv: "ac1ce190eddb9d41d9576c1a98e377dd"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.284101 9514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:10.321761 9514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:10.322715 9514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:10.322902 9514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:10.330924 9514 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:33735
I20260808 08:24:10.330948 9602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:33735 every 8 connection(s)
I20260808 08:24:10.331979 9514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:10.332525 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 9514
I20260808 08:24:10.332675 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:10.332988 12340 external_mini_cluster.cc:1428] Setting key 2ab387433571a6c893c162d87f58ad94
I20260808 08:24:10.335646 9603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:10.341662 9603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4: Bootstrap starting.
I20260808 08:24:10.342912 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:10.334493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.12.13.62:33735)
0808 08:24:10.335244 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:10.335268 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:10.335332 (+ 64us) server_negotiation.cc:407] Connection header received
0808 08:24:10.336111 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:10.336131 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:10.336552 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:10.336903 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:10.337362 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.338462 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.339363 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.339873 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.340048 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:10.341073 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:10.341078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:10.341091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:10.341504 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:10.341563 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:10.341582 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:10.341694 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:10.341902 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 08:24:10.342172 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":156,"threads_started":1}
I20260808 08:24:10.344408 9603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:10.345270 9603 log.cc:824] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:10.347525 9603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4: No bootstrap required, opened a new log
I20260808 08:24:10.351317 9603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298fa819555c4010b20ee44b22e595a4" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33735 } }
I20260808 08:24:10.351599 9603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:10.351640 9603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298fa819555c4010b20ee44b22e595a4, State: Initialized, Role: FOLLOWER
I20260808 08:24:10.352093 9603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298fa819555c4010b20ee44b22e595a4" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33735 } }
I20260808 08:24:10.352227 9603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:10.352279 9603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:10.352355 9603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:10.353296 9603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298fa819555c4010b20ee44b22e595a4" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33735 } }
I20260808 08:24:10.353605 9603 leader_election.cc:302] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298fa819555c4010b20ee44b22e595a4; no voters:
I20260808 08:24:10.353946 9603 leader_election.cc:288] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:10.354063 9610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:10.354317 9610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [term 1 LEADER]: Becoming Leader. State: Replica: 298fa819555c4010b20ee44b22e595a4, State: Running, Role: LEADER
I20260808 08:24:10.354663 9610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298fa819555c4010b20ee44b22e595a4" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33735 } }
I20260808 08:24:10.355345 9603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:10.356454 9612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298fa819555c4010b20ee44b22e595a4. Latest consensus state: current_term: 1 leader_uuid: "298fa819555c4010b20ee44b22e595a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298fa819555c4010b20ee44b22e595a4" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33735 } } }
I20260808 08:24:10.356467 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298fa819555c4010b20ee44b22e595a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298fa819555c4010b20ee44b22e595a4" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33735 } } }
I20260808 08:24:10.356592 9612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:10.356612 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:10.357038 9623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:10.360430 9623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:10.366532 9623 catalog_manager.cc:1347] Generated new cluster ID: d7f3bb2838144b00ad7e2421b6437782
I20260808 08:24:10.366632 9623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:10.385999 9623 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:10.387452 9623 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:10.393153 9623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4: Generated new TSK 0
I20260808 08:24:10.394078 9623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:10.400593 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.1:0
--local_ip_for_outbound_sockets=127.12.13.1
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33735
--builtin_ntp_servers=127.12.13.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 08:24:10.523082 9636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:10.523368 9636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:10.523409 9636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:10.527453 9636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:10.527542 9636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:10.527601 9636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.1
I20260808 08:24:10.532409 9636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.13.1
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.1
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:10.533542 9636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:10.534982 9636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:10.545904 9645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:10.545904 9642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:10.545904 9643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:10.546159 9636 server_base.cc:1034] running on GCE node
I20260808 08:24:10.546823 9636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:10.547547 9636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:10.548744 9636 hybrid_clock.cc:648] HybridClock initialized: now 1786177450548715 us; error 64 us; skew 500 ppm
I20260808 08:24:10.550758 9636 webserver.cc:460] Webserver started at http://127.12.13.1:32985/ using document root <none> and password file <none>
I20260808 08:24:10.551396 9636 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:10.551496 9636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:10.551748 9636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:10.554059 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data/instance:
uuid: "84c07a2aa62c4f77a47be81ace91454d"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "668dc9e296a2981f6e838d63c64bbe5a"
server_key_iv: "f4c2b3c369f11b7827cb297ed8d5c2b0"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.554673 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance:
uuid: "84c07a2aa62c4f77a47be81ace91454d"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "668dc9e296a2981f6e838d63c64bbe5a"
server_key_iv: "f4c2b3c369f11b7827cb297ed8d5c2b0"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.558966 9636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:10.561636 9653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.563022 9636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:10.563175 9636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal
uuid: "84c07a2aa62c4f77a47be81ace91454d"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "668dc9e296a2981f6e838d63c64bbe5a"
server_key_iv: "f4c2b3c369f11b7827cb297ed8d5c2b0"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.563344 9636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:10.593007 9636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:10.593894 9636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:10.594111 9636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:10.594825 9636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:10.595862 9636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:10.595937 9636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.596011 9636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:10.596057 9636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.605681 9636 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.1:38677
I20260808 08:24:10.605710 9803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.1:38677 every 8 connection(s)
I20260808 08:24:10.606719 9636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/data/info.pb
I20260808 08:24:10.608937 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 9636
I20260808 08:24:10.609074 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-0/wal/instance
I20260808 08:24:10.609390 12340 external_mini_cluster.cc:1428] Setting key 4ca7e3c8bc88b23544a9a749ec619470
I20260808 08:24:10.611758 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.2:0
--local_ip_for_outbound_sockets=127.12.13.2
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33735
--builtin_ntp_servers=127.12.13.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 08:24:10.618247 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:10.608753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.1:39033 (local address 127.12.13.62:33735)
0808 08:24:10.608929 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:10.608933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:10.610333 (+ 1400us) server_negotiation.cc:407] Connection header received
0808 08:24:10.612238 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:10.612241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:10.612300 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:10.612374 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:10.613690 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.614596 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.615765 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.616177 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.616299 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:10.617537 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:10.617543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:10.617548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:10.617569 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:10.617599 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:10.617612 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:10.617698 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:10.617905 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 08:24:10.618016 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 08:24:10.619143 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:10.609274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33735 (local address 127.12.13.1:39033)
0808 08:24:10.610164 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:10.610222 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:10.611969 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:10.612608 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:10.612620 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:10.612738 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:10.613507 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:10.613530 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.614736 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.614739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:10.615631 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:10.615641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.616360 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.616363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:10.616518 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:10.617230 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:10.617232 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:10.617277 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:10.617362 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:10.617712 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:10.617720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:10.617724 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 08:24:10.617843 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 08:24:10.618163 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260808 08:24:10.620548 9804 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33735
I20260808 08:24:10.620842 9804 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:10.621420 9804 heartbeater.cc:511] Master 127.12.13.62:33735 requested a full tablet report, sending...
I20260808 08:24:10.623306 9556 ts_manager.cc:194] Registered new tserver with Master: 84c07a2aa62c4f77a47be81ace91454d (127.12.13.1:38677)
I20260808 08:24:10.625100 9556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.1:39033
W20260808 08:24:10.737068 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:10.737385 9807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:10.737426 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:10.741520 9807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:10.741578 9807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:10.741634 9807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.2
I20260808 08:24:10.746703 9807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.13.2
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.2
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:10.747947 9807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:10.749481 9807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:10.759963 9814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:10.759963 9820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:10.760254 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:10.760025 9807 server_base.cc:1034] running on GCE node
I20260808 08:24:10.760768 9807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:10.761473 9807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:10.762653 9807 hybrid_clock.cc:648] HybridClock initialized: now 1786177450762611 us; error 52 us; skew 500 ppm
I20260808 08:24:10.764763 9807 webserver.cc:460] Webserver started at http://127.12.13.2:33297/ using document root <none> and password file <none>
I20260808 08:24:10.765393 9807 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:10.765462 9807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:10.765720 9807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:10.768080 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data/instance:
uuid: "9a3977e7c31844e29080387674323a1a"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "3c4dd552dad6ae958cfd63b206333f4f"
server_key_iv: "a73f8cd627b935cd4e92c3c8a74a9ffe"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.768662 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance:
uuid: "9a3977e7c31844e29080387674323a1a"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "3c4dd552dad6ae958cfd63b206333f4f"
server_key_iv: "a73f8cd627b935cd4e92c3c8a74a9ffe"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.772552 9807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:10.775213 9829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.776600 9807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:24:10.776774 9807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal
uuid: "9a3977e7c31844e29080387674323a1a"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "3c4dd552dad6ae958cfd63b206333f4f"
server_key_iv: "a73f8cd627b935cd4e92c3c8a74a9ffe"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.776911 9807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:10.790644 9807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:10.791435 9807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:10.791661 9807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:10.792335 9807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:10.793398 9807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:10.793479 9807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.793558 9807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:10.793612 9807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.803645 9807 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.2:45611
I20260808 08:24:10.803671 9977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.2:45611 every 8 connection(s)
I20260808 08:24:10.804623 9807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/data/info.pb
I20260808 08:24:10.810679 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 9807
I20260808 08:24:10.810842 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal/instance
I20260808 08:24:10.811170 12340 external_mini_cluster.cc:1428] Setting key 1667ff78f0fc84bfa6d749982c191565
I20260808 08:24:10.813570 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.13.3:0
--local_ip_for_outbound_sockets=127.12.13.3
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33735
--builtin_ntp_servers=127.12.13.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 08:24:10.815212 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:10.806620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.2:35017 (local address 127.12.13.62:33735)
0808 08:24:10.806795 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:10.806800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:10.807890 (+ 1090us) server_negotiation.cc:407] Connection header received
0808 08:24:10.809348 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:10.809351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:10.809400 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:10.809478 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:10.810555 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.811358 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.812750 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.813050 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.813173 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:10.814550 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:10.814556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:10.814561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:10.814580 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:10.814608 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:10.814622 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:10.814708 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:10.814913 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 08:24:10.814990 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 08:24:10.816334 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:10.807058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33735 (local address 127.12.13.2:35017)
0808 08:24:10.807734 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:10.807777 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:10.809126 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:10.809656 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:10.809662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:10.809754 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:10.810401 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:10.810416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.811492 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.811495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:10.812619 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:10.812631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:10.813201 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:10.813204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:10.813395 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:10.814278 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:10.814279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:10.814328 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:10.814393 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:10.814744 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:10.814754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:10.814760 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 08:24:10.814890 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 08:24:10.815148 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260808 08:24:10.817631 9978 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33735
I20260808 08:24:10.817950 9978 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:10.818547 9978 heartbeater.cc:511] Master 127.12.13.62:33735 requested a full tablet report, sending...
I20260808 08:24:10.819796 9556 ts_manager.cc:194] Registered new tserver with Master: 9a3977e7c31844e29080387674323a1a (127.12.13.2:45611)
I20260808 08:24:10.820690 9556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.2:35017
W20260808 08:24:10.936849 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:10.937153 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:10.937194 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:10.941448 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:10.941517 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:10.941574 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.13.3
I20260808 08:24:10.946666 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.13.3
--webserver_port=0
--tserver_master_addrs=127.12.13.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.13.3
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:10.947849 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:10.949277 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:10.960155 9988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:10.960166 9989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:10.960220 9981 server_base.cc:1034] running on GCE node
W20260808 08:24:10.960166 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:10.960850 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:10.961546 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:10.962731 9981 hybrid_clock.cc:648] HybridClock initialized: now 1786177450962684 us; error 61 us; skew 500 ppm
I20260808 08:24:10.964766 9981 webserver.cc:460] Webserver started at http://127.12.13.3:39145/ using document root <none> and password file <none>
I20260808 08:24:10.965399 9981 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:10.965497 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:10.965739 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:10.968080 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data/instance:
uuid: "99750cdcd195426bbe1399bd0b341fc5"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "47be73898ecf452112f41b5c67f847c9"
server_key_iv: "f50b8f0028c048c6fd437f4e1fee6b80"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.968710 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance:
uuid: "99750cdcd195426bbe1399bd0b341fc5"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "47be73898ecf452112f41b5c67f847c9"
server_key_iv: "f50b8f0028c048c6fd437f4e1fee6b80"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.972771 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:10.975409 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:10.976619 9981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:24:10.976763 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal
uuid: "99750cdcd195426bbe1399bd0b341fc5"
format_stamp: "Formatted at 2026-08-08 08:24:10 on dist-test-slave-6ntk"
server_key: "47be73898ecf452112f41b5c67f847c9"
server_key_iv: "f50b8f0028c048c6fd437f4e1fee6b80"
server_key_version: "encryptionkey@0"
I20260808 08:24:10.976889 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:11.010704 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:11.011515 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:11.011739 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:11.012419 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 08:24:11.013502 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 08:24:11.013576 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:11.013657 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 08:24:11.013712 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:11.023198 9981 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.3:37995
I20260808 08:24:11.023304 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.3:37995 every 8 connection(s)
I20260808 08:24:11.024232 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/data/info.pb
I20260808 08:24:11.033051 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 9981
I20260808 08:24:11.033311 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal/instance
I20260808 08:24:11.033638 12340 external_mini_cluster.cc:1428] Setting key 6d9459a3a4e56f0b38de31764dd26de3
I20260808 08:24:11.034137 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.026295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:54059 (local address 127.12.13.62:33735)
0808 08:24:11.026452 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.026456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.027628 (+ 1172us) server_negotiation.cc:407] Connection header received
0808 08:24:11.029133 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.029137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.029202 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.029319 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:11.030340 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.030875 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.032043 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.032268 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.032392 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.033598 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:11.033602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:11.033607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:11.033625 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:11.033644 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:11.033652 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:11.033732 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.033920 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.033987 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 08:24:11.035243 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.026765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33735 (local address 127.12.13.3:54059)
0808 08:24:11.027466 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.027510 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:11.028898 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:11.029501 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:11.029508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:11.029599 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:11.030186 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.030200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.031018 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.031020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.031911 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.031920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.032440 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.032444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.032617 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.033297 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 08:24:11.033299 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:11.033341 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 08:24:11.033434 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:11.033734 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:11.033747 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:11.033754 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 08:24:11.033922 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 08:24:11.034223 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260808 08:24:11.036466 10144 heartbeater.cc:347] Connected to a master server at 127.12.13.62:33735
I20260808 08:24:11.036768 10144 heartbeater.cc:464] Registering TS with master...
I20260808 08:24:11.037384 10144 heartbeater.cc:511] Master 127.12.13.62:33735 requested a full tablet report, sending...
I20260808 08:24:11.038661 9556 ts_manager.cc:194] Registered new tserver with Master: 99750cdcd195426bbe1399bd0b341fc5 (127.12.13.3:37995)
I20260808 08:24:11.039384 9556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.13.3:54059
I20260808 08:24:11.048215 12340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 08:24:11.054404 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.050386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.12.13.62:33735)
0808 08:24:11.050549 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.050554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.050752 (+ 198us) server_negotiation.cc:407] Connection header received
0808 08:24:11.050944 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.050947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.050993 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.051078 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:11.051373 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.051891 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.052736 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.052951 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.053058 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.053921 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:11.053925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:11.053928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:11.053949 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:11.053967 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:11.053975 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:11.054058 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.054173 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.054227 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 08:24:11.057576 9556 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 08:24:11.060863 9556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 08:24:11.076146 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.069844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38677 (local address 127.0.0.1:33798)
0808 08:24:11.070751 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.070837 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:11.071100 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:11.071817 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:11.071819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:11.071835 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:11.071972 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.071977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.073053 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.073058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.074364 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.074375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.075490 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.075494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.075617 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.075689 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 08:24:11.075806 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 08:24:11.075944 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":30,"spinlock_wait_cycles":4480,"thread_start_us":118,"threads_started":1}
I20260808 08:24:11.076473 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.070218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.3:37995 (local address 127.0.0.1:42074)
0808 08:24:11.070813 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.070852 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:11.071001 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:11.071776 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:11.071781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:11.071804 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:11.072014 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.072023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.073519 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.073522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.074636 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.074644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.075868 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.075871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.075963 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.075991 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 08:24:11.076064 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 08:24:11.076134 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":29,"spinlock_wait_cycles":12800,"thread_start_us":119,"threads_started":1}
I20260808 08:24:11.076522 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.070424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.12.13.3:37995)
0808 08:24:11.071058 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.071072 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.071113 (+ 41us) server_negotiation.cc:407] Connection header received
0808 08:24:11.071198 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.071203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.071426 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.071582 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:11.072147 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.073385 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.074766 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.075522 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.075616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.075798 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.076215 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.076342 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":104,"threads_started":1}
I20260808 08:24:11.076987 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.070006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.12.13.1:38677)
0808 08:24:11.070852 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.070860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.071066 (+ 206us) server_negotiation.cc:407] Connection header received
0808 08:24:11.071264 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.071273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.071492 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.071645 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:11.072129 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.072936 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.074510 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.075352 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.075474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.075969 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.076027 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.076139 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":152,"threads_started":1}
I20260808 08:24:11.077075 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.069844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45611 (local address 127.0.0.1:53248)
0808 08:24:11.071442 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.071455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:11.071549 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:11.072047 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:11.072052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:11.072072 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:11.072148 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.072153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.073998 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.074002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.075134 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.075144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.076574 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.076581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.076708 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.076739 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 08:24:11.076818 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 08:24:11.076935 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"mutex_wait_us":664,"thread_start_us":94,"threads_started":1}
I20260808 08:24:11.077600 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.070445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.12.13.2:45611)
0808 08:24:11.071306 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.071317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.071637 (+ 320us) server_negotiation.cc:407] Connection header received
0808 08:24:11.071719 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.071724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.071890 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.072050 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:11.072265 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.073391 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.075283 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.076165 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.076297 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.076500 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.077271 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.077407 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":120,"threads_started":1}
I20260808 08:24:11.080377 10062 tablet_service.cc:1467] Processing CreateTablet for tablet 333a0a491eb04ffbb3848daa3b930465 (DEFAULT_TABLE table=test-table [id=a5d01d3aca38479ca8db507b439d62a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:11.080377 9718 tablet_service.cc:1467] Processing CreateTablet for tablet 333a0a491eb04ffbb3848daa3b930465 (DEFAULT_TABLE table=test-table [id=a5d01d3aca38479ca8db507b439d62a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:11.080384 9890 tablet_service.cc:1467] Processing CreateTablet for tablet 333a0a491eb04ffbb3848daa3b930465 (DEFAULT_TABLE table=test-table [id=a5d01d3aca38479ca8db507b439d62a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 08:24:11.081734 9718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333a0a491eb04ffbb3848daa3b930465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:11.081735 10062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333a0a491eb04ffbb3848daa3b930465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:11.081734 9890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333a0a491eb04ffbb3848daa3b930465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:11.089592 10168 tablet_bootstrap.cc:492] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Bootstrap starting.
I20260808 08:24:11.091144 10169 tablet_bootstrap.cc:492] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: Bootstrap starting.
I20260808 08:24:11.092083 10168 tablet_bootstrap.cc:654] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:11.093063 10168 log.cc:824] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:11.093462 10169 tablet_bootstrap.cc:654] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:11.094290 10170 tablet_bootstrap.cc:492] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Bootstrap starting.
I20260808 08:24:11.094457 10169 log.cc:824] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:11.095434 10168 tablet_bootstrap.cc:492] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: No bootstrap required, opened a new log
I20260808 08:24:11.095657 10168 ts_tablet_manager.cc:1397] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:11.096745 10169 tablet_bootstrap.cc:492] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: No bootstrap required, opened a new log
I20260808 08:24:11.096865 10170 tablet_bootstrap.cc:654] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:11.096978 10169 ts_tablet_manager.cc:1397] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:11.097807 10170 log.cc:824] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:11.099507 10168 raft_consensus.cc:348] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.099813 10168 raft_consensus.cc:374] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:11.099881 10168 raft_consensus.cc:729] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3977e7c31844e29080387674323a1a, State: Initialized, Role: FOLLOWER
I20260808 08:24:11.100217 10170 tablet_bootstrap.cc:492] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: No bootstrap required, opened a new log
I20260808 08:24:11.100435 10170 ts_tablet_manager.cc:1397] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 08:24:11.100493 10168 consensus_queue.cc:260] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.101477 10168 ts_tablet_manager.cc:1428] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 08:24:11.101926 9978 heartbeater.cc:503] Master 127.12.13.62:33735 was elected leader, sending a full tablet report...
I20260808 08:24:11.102128 10169 raft_consensus.cc:348] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.102581 10169 raft_consensus.cc:374] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:11.102638 10169 raft_consensus.cc:729] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c07a2aa62c4f77a47be81ace91454d, State: Initialized, Role: FOLLOWER
I20260808 08:24:11.103276 10169 consensus_queue.cc:260] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.104069 9804 heartbeater.cc:503] Master 127.12.13.62:33735 was elected leader, sending a full tablet report...
I20260808 08:24:11.104485 10169 ts_tablet_manager.cc:1428] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 08:24:11.105330 10170 raft_consensus.cc:348] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.105654 10170 raft_consensus.cc:374] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:11.105737 10170 raft_consensus.cc:729] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99750cdcd195426bbe1399bd0b341fc5, State: Initialized, Role: FOLLOWER
I20260808 08:24:11.106348 10170 consensus_queue.cc:260] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.107293 10144 heartbeater.cc:503] Master 127.12.13.62:33735 was elected leader, sending a full tablet report...
I20260808 08:24:11.107481 10170 ts_tablet_manager.cc:1428] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260808 08:24:11.107980 9805 tablet.cc:2369] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:24:11.275348 10145 tablet.cc:2369] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 08:24:11.305996 9979 tablet.cc:2369] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:24:11.342474 10178 raft_consensus.cc:482] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 08:24:11.342738 10178 raft_consensus.cc:504] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.343993 10178 leader_election.cc:288] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c07a2aa62c4f77a47be81ace91454d (127.12.13.1:38677), 9a3977e7c31844e29080387674323a1a (127.12.13.2:45611)
I20260808 08:24:11.348730 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.344436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:33109 (local address 127.12.13.2:45611)
0808 08:24:11.344574 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.344579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.344597 (+ 18us) server_negotiation.cc:407] Connection header received
0808 08:24:11.344709 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.344715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.344771 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.344845 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:11.345197 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.346060 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.347500 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.348045 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.348178 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.348249 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.348498 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.348583 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 08:24:11.348711 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.344287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.2:45611 (local address 127.12.13.3:33109)
0808 08:24:11.344463 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.344482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:11.344597 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:11.344886 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:11.344890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:11.344911 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:11.345062 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.345071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.346166 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.346169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.347373 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.347384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.348218 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.348225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.348334 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.348356 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 08:24:11.348454 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 08:24:11.348537 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 08:24:11.348901 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.344566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.13.3:58241 (local address 127.12.13.1:38677)
0808 08:24:11.344693 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.344697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.345003 (+ 306us) server_negotiation.cc:407] Connection header received
0808 08:24:11.345107 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.345109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.345183 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.345264 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 08:24:11.345527 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.346380 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.347717 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.348319 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.348413 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.348506 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.348673 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.348753 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:24:11.349124 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.344349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38677 (local address 127.12.13.3:58241)
0808 08:24:11.344902 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.344915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:11.345005 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:11.345290 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:11.345293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:11.345312 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 08:24:11.345414 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.345418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.346510 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.346513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.347596 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:11.347603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.348446 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.348452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:11.348552 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.348572 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 08:24:11.348644 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 08:24:11.348696 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":117,"thread_start_us":133,"threads_started":1}
I20260808 08:24:11.349586 9916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a0a491eb04ffbb3848daa3b930465" candidate_uuid: "99750cdcd195426bbe1399bd0b341fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3977e7c31844e29080387674323a1a" is_pre_election: true
I20260808 08:24:11.349812 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a0a491eb04ffbb3848daa3b930465" candidate_uuid: "99750cdcd195426bbe1399bd0b341fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c07a2aa62c4f77a47be81ace91454d" is_pre_election: true
I20260808 08:24:11.349979 9916 raft_consensus.cc:2454] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99750cdcd195426bbe1399bd0b341fc5 in term 0.
I20260808 08:24:11.350168 9744 raft_consensus.cc:2454] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99750cdcd195426bbe1399bd0b341fc5 in term 0.
I20260808 08:24:11.350582 10003 leader_election.cc:302] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99750cdcd195426bbe1399bd0b341fc5, 9a3977e7c31844e29080387674323a1a; no voters:
I20260808 08:24:11.350901 10178 raft_consensus.cc:2781] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 08:24:11.350986 10178 raft_consensus.cc:482] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 08:24:11.351017 10178 raft_consensus.cc:3037] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:11.352128 10178 raft_consensus.cc:504] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.352550 10178 leader_election.cc:288] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [CANDIDATE]: Term 1 election: Requested vote from peers 84c07a2aa62c4f77a47be81ace91454d (127.12.13.1:38677), 9a3977e7c31844e29080387674323a1a (127.12.13.2:45611)
I20260808 08:24:11.352905 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a0a491eb04ffbb3848daa3b930465" candidate_uuid: "99750cdcd195426bbe1399bd0b341fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c07a2aa62c4f77a47be81ace91454d"
I20260808 08:24:11.352977 9916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a0a491eb04ffbb3848daa3b930465" candidate_uuid: "99750cdcd195426bbe1399bd0b341fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3977e7c31844e29080387674323a1a"
I20260808 08:24:11.353027 9744 raft_consensus.cc:3037] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:11.353140 9916 raft_consensus.cc:3037] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:11.354271 9744 raft_consensus.cc:2454] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99750cdcd195426bbe1399bd0b341fc5 in term 1.
I20260808 08:24:11.354447 9916 raft_consensus.cc:2454] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99750cdcd195426bbe1399bd0b341fc5 in term 1.
I20260808 08:24:11.354656 10001 leader_election.cc:302] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c07a2aa62c4f77a47be81ace91454d, 99750cdcd195426bbe1399bd0b341fc5; no voters:
I20260808 08:24:11.354851 10178 raft_consensus.cc:2781] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:11.355110 10178 raft_consensus.cc:686] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 1 LEADER]: Becoming Leader. State: Replica: 99750cdcd195426bbe1399bd0b341fc5, State: Running, Role: LEADER
I20260808 08:24:11.355520 10178 consensus_queue.cc:237] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } }
I20260808 08:24:11.359073 9556 catalog_manager.cc:5495] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99750cdcd195426bbe1399bd0b341fc5 (127.12.13.3). New cstate: current_term: 1 leader_uuid: "99750cdcd195426bbe1399bd0b341fc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99750cdcd195426bbe1399bd0b341fc5" member_type: VOTER last_known_addr { host: "127.12.13.3" port: 37995 } health_report { overall_health: HEALTHY } } }
I20260808 08:24:11.371826 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.368057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.12.13.3:37995)
0808 08:24:11.368187 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:11.368191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:11.368247 (+ 56us) server_negotiation.cc:407] Connection header received
0808 08:24:11.368424 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:11.368427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:11.368488 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:11.368581 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:11.368824 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.369362 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.370342 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:11.370545 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:11.370624 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:11.370973 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:11.371435 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:11.371538 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:11.371632 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 08:24:11.371681 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:24:11.378175 9916 raft_consensus.cc:1261] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 1 FOLLOWER]: Refusing update from remote peer 99750cdcd195426bbe1399bd0b341fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:11.378176 9744 raft_consensus.cc:1261] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d [term 1 FOLLOWER]: Refusing update from remote peer 99750cdcd195426bbe1399bd0b341fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:11.378947 10178 consensus_queue.cc:1035] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c07a2aa62c4f77a47be81ace91454d" member_type: VOTER last_known_addr { host: "127.12.13.1" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:11.379177 10180 consensus_queue.cc:1035] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3977e7c31844e29080387674323a1a" member_type: VOTER last_known_addr { host: "127.12.13.2" port: 45611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:11.399101 9556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:11.399298 9556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 08:24:11.401893 9556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298fa819555c4010b20ee44b22e595a4: Sending DeleteTablet for 3 replicas of tablet 333a0a491eb04ffbb3848daa3b930465
I20260808 08:24:11.402863 9890 tablet_service.cc:1514] Processing DeleteTablet for tablet 333a0a491eb04ffbb3848daa3b930465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:11 UTC) from {username='slave'} at 127.0.0.1:53248
I20260808 08:24:11.402920 10062 tablet_service.cc:1514] Processing DeleteTablet for tablet 333a0a491eb04ffbb3848daa3b930465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:11 UTC) from {username='slave'} at 127.0.0.1:42074
I20260808 08:24:11.403112 9718 tablet_service.cc:1514] Processing DeleteTablet for tablet 333a0a491eb04ffbb3848daa3b930465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 08:24:11 UTC) from {username='slave'} at 127.0.0.1:33798
I20260808 08:24:11.403460 10197 tablet_replica.cc:330] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: stopping tablet replica
I20260808 08:24:11.403465 10198 tablet_replica.cc:330] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: stopping tablet replica
I20260808 08:24:11.403796 10197 raft_consensus.cc:2229] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 08:24:11.403810 10198 raft_consensus.cc:2229] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 08:24:11.404240 10197 raft_consensus.cc:2258] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:11.404475 10199 tablet_replica.cc:330] T 333a0a491eb04ffbb3848daa3b930465 P 84c07a2aa62c4f77a47be81ace91454d: stopping tablet replica
I20260808 08:24:11.404502 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 9636
I20260808 08:24:11.410436 10197 ts_tablet_manager.cc:1905] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:11.413904 10197 ts_tablet_manager.cc:1918] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:11.414106 10197 log.cc:1196] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-2/wal/wals/333a0a491eb04ffbb3848daa3b930465
I20260808 08:24:11.414486 10197 ts_tablet_manager.cc:1939] T 333a0a491eb04ffbb3848daa3b930465 P 99750cdcd195426bbe1399bd0b341fc5: Deleting consensus metadata
I20260808 08:24:11.415848 9539 catalog_manager.cc:4841] TS 99750cdcd195426bbe1399bd0b341fc5 (127.12.13.3:37995): tablet 333a0a491eb04ffbb3848daa3b930465 (table test-table [id=a5d01d3aca38479ca8db507b439d62a4]) successfully deleted
I20260808 08:24:11.404168 10198 raft_consensus.cc:2258] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 08:24:11.421500 10198 ts_tablet_manager.cc:1905] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 08:24:11.423883 10198 ts_tablet_manager.cc:1918] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 08:24:11.424026 10198 log.cc:1196] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Deleting WAL directory at /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786177179845590-12340-0/minicluster-data/ts-1/wal/wals/333a0a491eb04ffbb3848daa3b930465
I20260808 08:24:11.424343 10198 ts_tablet_manager.cc:1939] T 333a0a491eb04ffbb3848daa3b930465 P 9a3977e7c31844e29080387674323a1a: Deleting consensus metadata
I20260808 08:24:11.425608 9540 catalog_manager.cc:4841] TS 9a3977e7c31844e29080387674323a1a (127.12.13.2:45611): tablet 333a0a491eb04ffbb3848daa3b930465 (table test-table [id=a5d01d3aca38479ca8db507b439d62a4]) successfully deleted
W20260808 08:24:11.427985 9538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.13.1:38677 (error 108)
I20260808 08:24:11.428941 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 9807
I20260808 08:24:11.429116 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:11.428810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.1:38677 (local address 127.0.0.1:51240)
0808 08:24:11.428952 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:11.429027 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.1:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 08:24:11.435889 9538 catalog_manager.cc:4584] TS 84c07a2aa62c4f77a47be81ace91454d (127.12.13.1:38677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 333a0a491eb04ffbb3848daa3b930465: Network error: Client connection negotiation failed: client connection to 127.12.13.1:38677: connect: Connection refused (error 111)
I20260808 08:24:11.462731 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 9981
I20260808 08:24:11.479866 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 9514
2026-08-08T08:24:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1420 ms)
[----------] 6 tests from EncryptionPolicyTest (11762 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:11 dist-test-slave-6ntk krb5kdc[10205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:11 dist-test-slave-6ntk krb5kdc[10205](info): set up 2 sockets
Aug 08 08:24:11 dist-test-slave-6ntk krb5kdc[10205](info): commencing operation
krb5kdc: starting...
W20260808 08:24:12.735513 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:12 dist-test-slave-6ntk krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:12.874055 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:12.995343 10221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:12.995716 10221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:12.995810 10221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:13.000243 10221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:13.000358 10221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:13.000397 10221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:13.000437 10221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:13.000484 10221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:13.006027 10221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:13.007266 10221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:13.008487 10221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:13.018102 10227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:13.018167 10221 server_base.cc:1034] running on GCE node
W20260808 08:24:13.018242 10228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:13.018193 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:13.019243 10221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:13.020430 10221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:13.021620 10221 hybrid_clock.cc:648] HybridClock initialized: now 1786177453021588 us; error 43 us; skew 500 ppm
Aug 08 08:24:13 dist-test-slave-6ntk krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:13.024654 10221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:13.025648 10221 webserver.cc:460] Webserver started at http://127.0.0.1:41359/ using document root <none> and password file <none>
I20260808 08:24:13.026345 10221 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:13.026451 10221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:13.026706 10221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:13.028893 10221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "49d211f5e39a4c1a88283efc93f08014"
format_stamp: "Formatted at 2026-08-08 08:24:13 on dist-test-slave-6ntk"
server_key: "f87033fbd2d20a2a9d968b7e29b7b966"
server_key_iv: "7f1d15530e46dc8540a68ed59ca6cf03"
server_key_version: "encryptionkey@0"
I20260808 08:24:13.029492 10221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "49d211f5e39a4c1a88283efc93f08014"
format_stamp: "Formatted at 2026-08-08 08:24:13 on dist-test-slave-6ntk"
server_key: "f87033fbd2d20a2a9d968b7e29b7b966"
server_key_iv: "7f1d15530e46dc8540a68ed59ca6cf03"
server_key_version: "encryptionkey@0"
I20260808 08:24:13.033454 10221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:13.036198 10239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:13.037597 10221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 08:24:13.037760 10221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "49d211f5e39a4c1a88283efc93f08014"
format_stamp: "Formatted at 2026-08-08 08:24:13 on dist-test-slave-6ntk"
server_key: "f87033fbd2d20a2a9d968b7e29b7b966"
server_key_iv: "7f1d15530e46dc8540a68ed59ca6cf03"
server_key_version: "encryptionkey@0"
I20260808 08:24:13.037922 10221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:13.066248 10221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:13.070000 10221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:13.070235 10221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:13.078265 10221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46113
I20260808 08:24:13.078290 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46113 every 8 connection(s)
I20260808 08:24:13.079334 10221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:13.082305 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 10221
I20260808 08:24:13.082442 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:13.082683 10306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:13.082770 12340 external_mini_cluster.cc:1428] Setting key d25a19d1f8f82000b7bca154039d934c
I20260808 08:24:13.089854 10306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014: Bootstrap starting.
Aug 08 08:24:13 dist-test-slave-6ntk krb5kdc[10205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 08:24:13.092655 10306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:13.093480 10306 log.cc:824] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:13.096182 10306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014: No bootstrap required, opened a new log
I20260808 08:24:13.098855 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:13.084441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57335 (local address 127.0.0.1:46113)
0808 08:24:13.084947 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:13.084963 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:13.085000 (+ 37us) server_negotiation.cc:407] Connection header received
0808 08:24:13.085668 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:13.085686 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:13.086155 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:13.086518 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:13.087412 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:13.088460 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:13.089363 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:13.089859 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:13.090100 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:13.093056 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:13.093062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:13.093077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:13.093108 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:13.095368 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:13.095929 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:13.095937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:13.095943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:13.096087 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:13.096485 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:13.096489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:13.096490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:13.097025 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:13.097234 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:13.097494 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 08:24:13.097727 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":173,"threads_started":1}
I20260808 08:24:13.101078 10306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d211f5e39a4c1a88283efc93f08014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46113 } }
I20260808 08:24:13.101398 10306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:13.101462 10306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d211f5e39a4c1a88283efc93f08014, State: Initialized, Role: FOLLOWER
I20260808 08:24:13.102023 10306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d211f5e39a4c1a88283efc93f08014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46113 } }
I20260808 08:24:13.102201 10306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:13.102286 10306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:13.102348 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:13.102394 10306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:13.103406 10306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d211f5e39a4c1a88283efc93f08014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46113 } }
I20260808 08:24:13.103828 10306 leader_election.cc:302] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d211f5e39a4c1a88283efc93f08014; no voters:
I20260808 08:24:13.104226 10306 leader_election.cc:288] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:13.104302 10321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:13.104506 10321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [term 1 LEADER]: Becoming Leader. State: Replica: 49d211f5e39a4c1a88283efc93f08014, State: Running, Role: LEADER
I20260808 08:24:13.104816 10321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d211f5e39a4c1a88283efc93f08014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46113 } }
I20260808 08:24:13.105664 10306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:13.106110 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d211f5e39a4c1a88283efc93f08014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d211f5e39a4c1a88283efc93f08014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46113 } } }
I20260808 08:24:13.106230 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:13.106458 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d211f5e39a4c1a88283efc93f08014. Latest consensus state: current_term: 1 leader_uuid: "49d211f5e39a4c1a88283efc93f08014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d211f5e39a4c1a88283efc93f08014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46113 } } }
I20260808 08:24:13.106562 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:13.106945 10327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:13.110737 10327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:13.114277 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:13.104563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52877 (local address 127.0.0.1:46113)
0808 08:24:13.104707 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:13.104713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:13.105047 (+ 334us) server_negotiation.cc:407] Connection header received
0808 08:24:13.105210 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:13.105213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:13.105269 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:13.105357 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:13.106223 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:13.107008 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:13.108196 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:13.108442 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:13.108562 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:13.110332 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:13.110335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:13.110338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:13.110367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:13.111885 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:13.112670 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:13.112673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:13.112675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:13.112801 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:13.113215 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:13.113219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:13.113221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:13.113470 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:13.113630 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:13.114002 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 08:24:13.114118 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 08:24:13.117139 10327 catalog_manager.cc:1347] Generated new cluster ID: b3a86b573de540369f14c43195f449df
I20260808 08:24:13.117216 10327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:13.133307 10327 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:13.134397 10327 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:13.149260 10327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d211f5e39a4c1a88283efc93f08014: Generated new TSK 0
I20260808 08:24:13.150357 10327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:13.164287 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 10221
W20260808 08:24:13.181285 10203 connection.cc:535] client connection to 127.0.0.1:46113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46113): Connection reset by peer (error 104)
2026-08-08T08:24:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1680 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:13 dist-test-slave-6ntk krb5kdc[10348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:13 dist-test-slave-6ntk krb5kdc[10348](info): set up 2 sockets
Aug 08 08:24:13 dist-test-slave-6ntk krb5kdc[10348](info): commencing operation
krb5kdc: starting...
W20260808 08:24:14.441601 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:14 dist-test-slave-6ntk krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:14.582501 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:14.701689 10369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:14.702028 10369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:14.702121 10369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:14.706189 10369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:14.706305 10369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:14.706346 10369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:14.706387 10369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:14.706424 10369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:14.711843 10369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:14.713048 10369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:14.714188 10369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:14.723117 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:14.723194 10369 server_base.cc:1034] running on GCE node
W20260808 08:24:14.723138 10374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:14.723117 10376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:14.724104 10369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:14.725253 10369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:14.726449 10369 hybrid_clock.cc:648] HybridClock initialized: now 1786177454726440 us; error 39 us; skew 500 ppm
Aug 08 08:24:14 dist-test-slave-6ntk krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:14.729393 10369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:14.730399 10369 webserver.cc:460] Webserver started at http://127.0.0.1:46419/ using document root <none> and password file <none>
I20260808 08:24:14.730973 10369 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:14.731034 10369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:14.731305 10369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:14.733373 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "fe65fb89acdb4dadbcfb1a14518b9646"
format_stamp: "Formatted at 2026-08-08 08:24:14 on dist-test-slave-6ntk"
server_key: "bc9c227b699fbcd561bc79f9916ede8c"
server_key_iv: "8126e000b2ae610fed68e679ac4cf35d"
server_key_version: "encryptionkey@0"
I20260808 08:24:14.734030 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "fe65fb89acdb4dadbcfb1a14518b9646"
format_stamp: "Formatted at 2026-08-08 08:24:14 on dist-test-slave-6ntk"
server_key: "bc9c227b699fbcd561bc79f9916ede8c"
server_key_iv: "8126e000b2ae610fed68e679ac4cf35d"
server_key_version: "encryptionkey@0"
I20260808 08:24:14.737815 10369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:24:14.740587 10387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:14.742172 10369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:14.742329 10369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "fe65fb89acdb4dadbcfb1a14518b9646"
format_stamp: "Formatted at 2026-08-08 08:24:14 on dist-test-slave-6ntk"
server_key: "bc9c227b699fbcd561bc79f9916ede8c"
server_key_iv: "8126e000b2ae610fed68e679ac4cf35d"
server_key_version: "encryptionkey@0"
I20260808 08:24:14.742450 10369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:14.756168 10369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:14.760212 10369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:14.760462 10369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:14.768847 10369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46527
I20260808 08:24:14.768878 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46527 every 8 connection(s)
I20260808 08:24:14.770017 10369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:14.770359 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 10369
I20260808 08:24:14.770543 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:14.770915 12340 external_mini_cluster.cc:1428] Setting key 96b6085143b596ff4b9653d3bb44f4a6
I20260808 08:24:14.773521 10458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:14.780153 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646: Bootstrap starting.
I20260808 08:24:14.780673 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:14.772364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40701 (local address 127.0.0.1:46527)
0808 08:24:14.772819 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:14.772842 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:14.773080 (+ 238us) server_negotiation.cc:407] Connection header received
0808 08:24:14.774014 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:14.774034 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:14.774398 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:14.774753 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:14.775189 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:14.776185 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:14.777101 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:14.777589 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:14.777784 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:14.778826 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:14.778831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:14.778844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:14.779247 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:14.779304 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:14.779327 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:14.779481 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:14.779689 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 08:24:14.779839 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":145,"threads_started":1}
I20260808 08:24:14.783022 10458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:14.784040 10458 log.cc:824] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:14.784153 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:14.786392 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646: No bootstrap required, opened a new log
I20260808 08:24:14.791395 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:14.786551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48139 (local address 127.0.0.1:46527)
0808 08:24:14.786698 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:14.786703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:14.786935 (+ 232us) server_negotiation.cc:407] Connection header received
0808 08:24:14.787058 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:14.787061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:14.787106 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:14.787200 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:14.787538 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:14.788066 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:14.789430 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:14.789641 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:14.789713 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:14.790741 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:14.790744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:14.790747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:14.790760 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:14.790788 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:14.790800 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:14.790897 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:14.791070 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 08:24:14.791208 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:24:14.791790 10458 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260808 08:24:14.792083 10458 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:14.792153 10458 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe65fb89acdb4dadbcfb1a14518b9646, State: Initialized, Role: FOLLOWER
I20260808 08:24:14.792713 10458 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260808 08:24:14.792878 10458 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:14.792956 10458 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:14.793087 10458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:14.794234 10458 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260808 08:24:14.794610 10458 leader_election.cc:302] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe65fb89acdb4dadbcfb1a14518b9646; no voters:
I20260808 08:24:14.794999 10458 leader_election.cc:288] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:14.795151 10475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:14.795406 10475 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [term 1 LEADER]: Becoming Leader. State: Replica: fe65fb89acdb4dadbcfb1a14518b9646, State: Running, Role: LEADER
I20260808 08:24:14.795825 10475 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260808 08:24:14.796521 10458 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:14.797276 10478 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe65fb89acdb4dadbcfb1a14518b9646. Latest consensus state: current_term: 1 leader_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } } }
I20260808 08:24:14.797497 10478 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:14.798496 10476 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe65fb89acdb4dadbcfb1a14518b9646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } } }
I20260808 08:24:14.798604 10476 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:14.799007 10487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:14.802700 10487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:14.814229 10487 catalog_manager.cc:1347] Generated new cluster ID: ec58ba61ee834f0c89bd5526a33def32
I20260808 08:24:14.814345 10487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:14.825084 10487 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:14.826195 10487 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:14.832262 10487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe65fb89acdb4dadbcfb1a14518b9646: Generated new TSK 0
I20260808 08:24:14.832980 10487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:14.838408 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 10369
W20260808 08:24:14.853083 10345 connection.cc:535] client connection to 127.0.0.1:46527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46527): Connection reset by peer (error 104)
2026-08-08T08:24:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1671 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:14 dist-test-slave-6ntk krb5kdc[10503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:14 dist-test-slave-6ntk krb5kdc[10503](info): set up 2 sockets
Aug 08 08:24:14 dist-test-slave-6ntk krb5kdc[10503](info): commencing operation
krb5kdc: starting...
W20260808 08:24:16.091002 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:16 dist-test-slave-6ntk krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:16.233091 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:16.352950 10517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:16.353253 10517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:16.353303 10517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:16.357342 10517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:16.357412 10517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:16.357432 10517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:16.357450 10517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:16.357465 10517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:16.362964 10517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:16.364143 10517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:16.365263 10517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:16.374411 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:16.374915 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:16.375025 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:16.375239 10517 server_base.cc:1034] running on GCE node
I20260808 08:24:16.376084 10517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:16.377327 10517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:16.378548 10517 hybrid_clock.cc:648] HybridClock initialized: now 1786177456378468 us; error 100 us; skew 500 ppm
Aug 08 08:24:16 dist-test-slave-6ntk krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:16.382645 10517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:16.384450 10517 webserver.cc:460] Webserver started at http://127.0.0.1:36783/ using document root <none> and password file <none>
I20260808 08:24:16.385298 10517 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:16.385392 10517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:16.385660 10517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:16.388245 10517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "b2214115fd674b65b0c36a87f433f312"
format_stamp: "Formatted at 2026-08-08 08:24:16 on dist-test-slave-6ntk"
server_key: "53103f2b328f4b8174b66c23c24d5986"
server_key_iv: "924d0c6465c2e928e6e1ef9a0256d702"
server_key_version: "encryptionkey@0"
I20260808 08:24:16.388898 10517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "b2214115fd674b65b0c36a87f433f312"
format_stamp: "Formatted at 2026-08-08 08:24:16 on dist-test-slave-6ntk"
server_key: "53103f2b328f4b8174b66c23c24d5986"
server_key_iv: "924d0c6465c2e928e6e1ef9a0256d702"
server_key_version: "encryptionkey@0"
I20260808 08:24:16.395550 10517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 08:24:16.399149 10536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:16.400894 10517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 08:24:16.401032 10517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "b2214115fd674b65b0c36a87f433f312"
format_stamp: "Formatted at 2026-08-08 08:24:16 on dist-test-slave-6ntk"
server_key: "53103f2b328f4b8174b66c23c24d5986"
server_key_iv: "924d0c6465c2e928e6e1ef9a0256d702"
server_key_version: "encryptionkey@0"
I20260808 08:24:16.401155 10517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:16.407611 10517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:16.411804 10517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:16.411980 10517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:16.421343 10517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38305
I20260808 08:24:16.421741 10606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38305 every 8 connection(s)
I20260808 08:24:16.422544 10517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:16.426679 10607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:16.430989 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 10517
I20260808 08:24:16.431144 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:16.431434 12340 external_mini_cluster.cc:1428] Setting key 793a150118a561ab5e9c4609e86773ac
I20260808 08:24:16.435940 10607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312: Bootstrap starting.
I20260808 08:24:16.438192 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:16.432890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60809 (local address 127.0.0.1:38305)
0808 08:24:16.433332 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:16.433351 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:16.433399 (+ 48us) server_negotiation.cc:407] Connection header received
0808 08:24:16.434366 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:16.434414 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:16.434835 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:16.435233 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:16.436851 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:16.436857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:16.436874 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:16.436894 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:16.436952 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:16.436972 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:16.437128 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:16.437200 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 08:24:16.437378 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":152,"threads_started":1}
I20260808 08:24:16.438879 10607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:16.439827 10607 log.cc:824] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:16.441386 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:16.442291 10607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312: No bootstrap required, opened a new log
I20260808 08:24:16.445559 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:16.443833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47897 (local address 127.0.0.1:38305)
0808 08:24:16.443955 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:16.443959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:16.444104 (+ 145us) server_negotiation.cc:407] Connection header received
0808 08:24:16.444238 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:16.444244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:16.444281 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:16.444368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:16.445126 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:16.445130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:16.445132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:16.445142 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:16.445165 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:16.445175 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:16.445268 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:16.445369 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 08:24:16.445429 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 08:24:16.446913 10607 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2214115fd674b65b0c36a87f433f312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260808 08:24:16.447139 10607 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:16.447173 10607 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2214115fd674b65b0c36a87f433f312, State: Initialized, Role: FOLLOWER
I20260808 08:24:16.447547 10607 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2214115fd674b65b0c36a87f433f312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260808 08:24:16.447667 10607 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:16.447721 10607 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:16.447799 10607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:16.448918 10607 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2214115fd674b65b0c36a87f433f312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260808 08:24:16.449319 10607 leader_election.cc:302] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2214115fd674b65b0c36a87f433f312; no voters:
I20260808 08:24:16.449652 10607 leader_election.cc:288] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:16.449797 10620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:16.450062 10620 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [term 1 LEADER]: Becoming Leader. State: Replica: b2214115fd674b65b0c36a87f433f312, State: Running, Role: LEADER
I20260808 08:24:16.450407 10620 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2214115fd674b65b0c36a87f433f312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260808 08:24:16.451012 10607 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:16.452138 10625 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2214115fd674b65b0c36a87f433f312. Latest consensus state: current_term: 1 leader_uuid: "b2214115fd674b65b0c36a87f433f312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2214115fd674b65b0c36a87f433f312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260808 08:24:16.452270 10625 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:16.452591 10623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2214115fd674b65b0c36a87f433f312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2214115fd674b65b0c36a87f433f312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260808 08:24:16.452698 10623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:16.452795 10636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:16.456169 10636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:16.461823 10636 catalog_manager.cc:1347] Generated new cluster ID: 19e8f9ba6d5141cb9fafd33172c17c38
I20260808 08:24:16.461922 10636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:16.486903 10636 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:16.487279 10636 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:16.502208 10636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2214115fd674b65b0c36a87f433f312: Generated new TSK 0
I20260808 08:24:16.503012 10636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:16.509352 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 10517
2026-08-08T08:24:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1673 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:16 dist-test-slave-6ntk krb5kdc[10650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:16 dist-test-slave-6ntk krb5kdc[10650](info): set up 2 sockets
Aug 08 08:24:16 dist-test-slave-6ntk krb5kdc[10650](info): commencing operation
krb5kdc: starting...
W20260808 08:24:17.803447 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:17 dist-test-slave-6ntk krb5kdc[10650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:17.937382 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:18.053828 10667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:18.054173 10667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:18.054248 10667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:18.058259 10667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:18.058359 10667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:18.058403 10667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:18.058456 10667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:18.058498 10667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:18.063834 10667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:18.064967 10667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:18.066080 10667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:18.074786 10672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:18.074782 10675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:18.074810 10673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:18.074838 10667 server_base.cc:1034] running on GCE node
I20260808 08:24:18.075775 10667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:18.076926 10667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:18.078112 10667 hybrid_clock.cc:648] HybridClock initialized: now 1786177458078083 us; error 43 us; skew 500 ppm
Aug 08 08:24:18 dist-test-slave-6ntk krb5kdc[10650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:18.081354 10667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:18.082370 10667 webserver.cc:460] Webserver started at http://127.0.0.1:36431/ using document root <none> and password file <none>
I20260808 08:24:18.083020 10667 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:18.083115 10667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:18.083335 10667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:18.085614 10667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "d5bb01a043a24502b8eebb29c37c1079"
format_stamp: "Formatted at 2026-08-08 08:24:18 on dist-test-slave-6ntk"
server_key: "aa289b8fe2592271ebf47d846cf895d5"
server_key_iv: "4ff50a6c79e2ad6eab875d602ee9a770"
server_key_version: "encryptionkey@0"
I20260808 08:24:18.086277 10667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "d5bb01a043a24502b8eebb29c37c1079"
format_stamp: "Formatted at 2026-08-08 08:24:18 on dist-test-slave-6ntk"
server_key: "aa289b8fe2592271ebf47d846cf895d5"
server_key_iv: "4ff50a6c79e2ad6eab875d602ee9a770"
server_key_version: "encryptionkey@0"
I20260808 08:24:18.090245 10667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:18.092898 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:18.094264 10667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:18.094473 10667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "d5bb01a043a24502b8eebb29c37c1079"
format_stamp: "Formatted at 2026-08-08 08:24:18 on dist-test-slave-6ntk"
server_key: "aa289b8fe2592271ebf47d846cf895d5"
server_key_iv: "4ff50a6c79e2ad6eab875d602ee9a770"
server_key_version: "encryptionkey@0"
I20260808 08:24:18.094599 10667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:18.114396 10667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:18.117568 10667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:18.117789 10667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:18.125340 10667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42385
I20260808 08:24:18.125377 10758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42385 every 8 connection(s)
I20260808 08:24:18.126453 10667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:18.129456 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:18.135331 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079: Bootstrap starting.
I20260808 08:24:18.135515 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 10667
I20260808 08:24:18.135679 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:18.135998 12340 external_mini_cluster.cc:1428] Setting key 8002b1a5c873085bc1de57ae46d2bfff
I20260808 08:24:18.138228 10759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:18.138991 10759 log.cc:824] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:18.141002 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079: No bootstrap required, opened a new log
I20260808 08:24:18.144843 10759 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bb01a043a24502b8eebb29c37c1079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260808 08:24:18.145102 10759 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 08 08:24:18 dist-test-slave-6ntk krb5kdc[10650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 08:24:18.145179 10759 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5bb01a043a24502b8eebb29c37c1079, State: Initialized, Role: FOLLOWER
I20260808 08:24:18.145738 10759 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bb01a043a24502b8eebb29c37c1079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260808 08:24:18.145975 10759 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:18.146067 10759 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:18.146170 10759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:18.147344 10759 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bb01a043a24502b8eebb29c37c1079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260808 08:24:18.147704 10759 leader_election.cc:302] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5bb01a043a24502b8eebb29c37c1079; no voters:
I20260808 08:24:18.148072 10759 leader_election.cc:288] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:18.148214 10764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:18.148463 10764 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [term 1 LEADER]: Becoming Leader. State: Replica: d5bb01a043a24502b8eebb29c37c1079, State: Running, Role: LEADER
I20260808 08:24:18.148829 10764 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bb01a043a24502b8eebb29c37c1079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260808 08:24:18.149549 10759 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:18.150630 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5bb01a043a24502b8eebb29c37c1079. Latest consensus state: current_term: 1 leader_uuid: "d5bb01a043a24502b8eebb29c37c1079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bb01a043a24502b8eebb29c37c1079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } } }
I20260808 08:24:18.150753 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:18.151080 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5bb01a043a24502b8eebb29c37c1079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bb01a043a24502b8eebb29c37c1079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } } }
I20260808 08:24:18.151158 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:18.151417 10774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:18.151683 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:18.137471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34433 (local address 127.0.0.1:42385)
0808 08:24:18.137955 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:18.137975 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:18.138014 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:24:18.138748 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:18.138768 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:18.139157 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:18.139520 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:18.140395 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:18.141457 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:18.142408 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:18.142879 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:18.143057 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:18.145848 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:18.145853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:18.145947 (+ 94us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:18.145998 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:18.147916 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:18.148463 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:18.148469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:18.148475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:18.148618 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:18.148971 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:18.148975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:18.148976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:18.149498 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:18.149729 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:18.150148 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 08:24:18.150551 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260808 08:24:18.155550 10774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:18.155905 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:18.162264 10774 catalog_manager.cc:1347] Generated new cluster ID: c051fa91b8f34a0fa07cb0e7552800dc
I20260808 08:24:18.162346 10774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:18.167475 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:18.158296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59699 (local address 127.0.0.1:42385)
0808 08:24:18.158438 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:18.158442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:18.158680 (+ 238us) server_negotiation.cc:407] Connection header received
0808 08:24:18.158819 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:18.158822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:18.158874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:18.158966 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:18.159662 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:18.160372 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:18.161497 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:18.161860 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:18.161996 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:18.163945 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:18.163949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:18.163952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:18.163978 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:18.165658 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:18.166209 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:18.166212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:18.166214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:18.166299 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:18.166586 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:18.166589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:18.166591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:18.166802 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:18.166922 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:18.167166 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 08:24:18.167280 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 08:24:18.173935 10774 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:18.175217 10774 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:18.190282 10774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5bb01a043a24502b8eebb29c37c1079: Generated new TSK 0
I20260808 08:24:18.191376 10774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:18.199041 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 10667
2026-08-08T08:24:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1689 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:18 dist-test-slave-6ntk krb5kdc[10798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:18 dist-test-slave-6ntk krb5kdc[10798](info): set up 2 sockets
Aug 08 08:24:18 dist-test-slave-6ntk krb5kdc[10798](info): commencing operation
krb5kdc: starting...
W20260808 08:24:19.466118 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:19 dist-test-slave-6ntk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:19.603215 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:19.723584 10815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:19.723875 10815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:19.723918 10815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:19.727896 10815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:19.727960 10815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:19.727988 10815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:19.728004 10815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:19.728020 10815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:19.733295 10815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:19.734452 10815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:19.735584 10815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:19.744975 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:19.745057 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:19.745102 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:19.745270 10815 server_base.cc:1034] running on GCE node
I20260808 08:24:19.746095 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:19.747296 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:19.748476 10815 hybrid_clock.cc:648] HybridClock initialized: now 1786177459748447 us; error 52 us; skew 500 ppm
Aug 08 08:24:19 dist-test-slave-6ntk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:19.751422 10815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:19.752427 10815 webserver.cc:460] Webserver started at http://127.0.0.1:35773/ using document root <none> and password file <none>
I20260808 08:24:19.753050 10815 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:19.753116 10815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:19.753364 10815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:19.755656 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "097f4187579c46b9a70552a301c42905"
format_stamp: "Formatted at 2026-08-08 08:24:19 on dist-test-slave-6ntk"
server_key: "e927e407cbf8f802222062f78d31d316"
server_key_iv: "2cb18f907316865235adf8de88c88e72"
server_key_version: "encryptionkey@0"
I20260808 08:24:19.756260 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "097f4187579c46b9a70552a301c42905"
format_stamp: "Formatted at 2026-08-08 08:24:19 on dist-test-slave-6ntk"
server_key: "e927e407cbf8f802222062f78d31d316"
server_key_iv: "2cb18f907316865235adf8de88c88e72"
server_key_version: "encryptionkey@0"
I20260808 08:24:19.760310 10815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:19.762953 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:19.764272 10815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:19.764411 10815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "097f4187579c46b9a70552a301c42905"
format_stamp: "Formatted at 2026-08-08 08:24:19 on dist-test-slave-6ntk"
server_key: "e927e407cbf8f802222062f78d31d316"
server_key_iv: "2cb18f907316865235adf8de88c88e72"
server_key_version: "encryptionkey@0"
I20260808 08:24:19.764523 10815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:19.790638 10815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:19.794049 10815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:19.794276 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:19.802251 10815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46839
I20260808 08:24:19.802270 10903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46839 every 8 connection(s)
I20260808 08:24:19.803279 10815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:19.806326 10904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:19.811620 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 10815
I20260808 08:24:19.811813 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:19.812142 12340 external_mini_cluster.cc:1428] Setting key c30dce2de1d2d228080a48dda71bf93c
I20260808 08:24:19.812880 10904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905: Bootstrap starting.
I20260808 08:24:19.815469 10904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:19.816251 10904 log.cc:824] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:19.818274 10904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905: No bootstrap required, opened a new log
I20260808 08:24:19.821504 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:19.813595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35383 (local address 127.0.0.1:46839)
0808 08:24:19.814132 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:19.814149 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:19.814185 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:24:19.814889 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:19.814909 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:19.815298 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:19.815650 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:19.816097 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:19.817131 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:19.818130 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:19.818654 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:19.818830 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:19.819749 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:19.819754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:19.819766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:19.820176 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:19.820234 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:19.820257 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:19.820369 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:19.820519 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 08:24:19.820737 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":144,"threads_started":1}
I20260808 08:24:19.822414 10904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097f4187579c46b9a70552a301c42905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260808 08:24:19.822705 10904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:19.822785 10904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097f4187579c46b9a70552a301c42905, State: Initialized, Role: FOLLOWER
I20260808 08:24:19.823266 10904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097f4187579c46b9a70552a301c42905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260808 08:24:19.823396 10904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:19.823482 10904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:19.823587 10904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:19.824800 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:19.824774 10904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097f4187579c46b9a70552a301c42905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260808 08:24:19.825135 10904 leader_election.cc:302] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 097f4187579c46b9a70552a301c42905; no voters:
I20260808 08:24:19.825515 10904 leader_election.cc:288] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:19.825634 10909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:19.825932 10909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [term 1 LEADER]: Becoming Leader. State: Replica: 097f4187579c46b9a70552a301c42905, State: Running, Role: LEADER
I20260808 08:24:19.826262 10909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097f4187579c46b9a70552a301c42905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260808 08:24:19.827013 10904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:19.828184 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 097f4187579c46b9a70552a301c42905. Latest consensus state: current_term: 1 leader_uuid: "097f4187579c46b9a70552a301c42905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097f4187579c46b9a70552a301c42905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } } }
I20260808 08:24:19.828194 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "097f4187579c46b9a70552a301c42905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097f4187579c46b9a70552a301c42905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } } }
I20260808 08:24:19.828429 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:19.828461 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905 [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:19.832904 10932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:19.833005 10932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:19.833086 10924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:19.834221 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:19.827279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48131 (local address 127.0.0.1:46839)
0808 08:24:19.827425 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:19.827431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:19.828768 (+ 1337us) server_negotiation.cc:407] Connection header received
0808 08:24:19.828970 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:19.828976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:19.829028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:19.829082 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:19.830229 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:19.830977 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:19.832131 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:19.832466 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:19.832545 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:19.833642 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:19.833646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:19.833649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:19.833664 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:19.833695 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:19.833710 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:19.833807 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:19.834011 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 08:24:19.834073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 08:24:19.834257 10924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:19.839998 10924 catalog_manager.cc:1347] Generated new cluster ID: fd5fd52e98734e758d043ee97c28f452
I20260808 08:24:19.840085 10924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:19.870589 10924 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:19.871733 10924 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:19.879132 10924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 097f4187579c46b9a70552a301c42905: Generated new TSK 0
I20260808 08:24:19.879870 10924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:19.884161 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 10815
2026-08-08T08:24:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1680 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:19 dist-test-slave-6ntk krb5kdc[10939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:19 dist-test-slave-6ntk krb5kdc[10939](info): set up 2 sockets
Aug 08 08:24:19 dist-test-slave-6ntk krb5kdc[10939](info): commencing operation
krb5kdc: starting...
W20260808 08:24:21.142009 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:21 dist-test-slave-6ntk krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:21.280519 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:21.404420 10961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:21.404728 10961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:21.404774 10961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:21.409035 10961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:21.409114 10961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:21.409148 10961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:21.409214 10961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:21.409255 10961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:21.414901 10961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:21.416209 10961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:21.417496 10961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:21.426719 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:21.426760 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:21.426739 10972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:21.427197 10961 server_base.cc:1034] running on GCE node
I20260808 08:24:21.427834 10961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:21.429008 10961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:21.430169 10961 hybrid_clock.cc:648] HybridClock initialized: now 1786177461430138 us; error 45 us; skew 500 ppm
Aug 08 08:24:21 dist-test-slave-6ntk krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:21.434018 10961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:21.435047 10961 webserver.cc:460] Webserver started at http://127.0.0.1:38209/ using document root <none> and password file <none>
I20260808 08:24:21.435657 10961 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:21.435748 10961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:21.435973 10961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:21.438194 10961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "66f49b3297174224869f29b3a86f038f"
format_stamp: "Formatted at 2026-08-08 08:24:21 on dist-test-slave-6ntk"
server_key: "c2165f6062c5dd99bec2e60834c0660d"
server_key_iv: "6ce846cb4c4257fdeacc2f5ed124ecb6"
server_key_version: "encryptionkey@0"
I20260808 08:24:21.438791 10961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "66f49b3297174224869f29b3a86f038f"
format_stamp: "Formatted at 2026-08-08 08:24:21 on dist-test-slave-6ntk"
server_key: "c2165f6062c5dd99bec2e60834c0660d"
server_key_iv: "6ce846cb4c4257fdeacc2f5ed124ecb6"
server_key_version: "encryptionkey@0"
I20260808 08:24:21.443044 10961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:21.445710 10982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:21.447057 10961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 08:24:21.447203 10961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "66f49b3297174224869f29b3a86f038f"
format_stamp: "Formatted at 2026-08-08 08:24:21 on dist-test-slave-6ntk"
server_key: "c2165f6062c5dd99bec2e60834c0660d"
server_key_iv: "6ce846cb4c4257fdeacc2f5ed124ecb6"
server_key_version: "encryptionkey@0"
I20260808 08:24:21.447330 10961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:21.452200 10961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:21.455384 10961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:21.455587 10961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:21.463311 10961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44431
I20260808 08:24:21.463336 11049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44431 every 8 connection(s)
I20260808 08:24:21.464323 10961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:21.467355 11050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:21.467928 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 10961
I20260808 08:24:21.468103 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:21.468487 12340 external_mini_cluster.cc:1428] Setting key e83c754a48eff7b394e8cc221eea4c27
I20260808 08:24:21.473975 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:21.470037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48829 (local address 127.0.0.1:44431)
0808 08:24:21.470504 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:21.470519 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:21.470558 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:24:21.471250 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:21.471267 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:21.471562 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:21.471963 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:21.472726 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:21.472731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:21.472744 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:21.472756 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:21.472816 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:21.472836 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:21.472987 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:21.473058 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 08:24:21.473217 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":129,"threads_started":1}
I20260808 08:24:21.474820 11050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f: Bootstrap starting.
I20260808 08:24:21.476881 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:21.477547 11050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:21.478406 11050 log.cc:824] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:21.480486 11050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f: No bootstrap required, opened a new log
I20260808 08:24:21.480837 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:21.479107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36523 (local address 127.0.0.1:44431)
0808 08:24:21.479233 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:21.479238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:21.479475 (+ 237us) server_negotiation.cc:407] Connection header received
0808 08:24:21.479617 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:21.479620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:21.479657 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:21.479734 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:21.480461 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:21.480465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:21.480468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:21.480479 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:21.480510 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:21.480520 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:21.480604 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:21.480680 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 08:24:21.480717 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:24:21.484401 11050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f49b3297174224869f29b3a86f038f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260808 08:24:21.484625 11050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:21.484699 11050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f49b3297174224869f29b3a86f038f, State: Initialized, Role: FOLLOWER
I20260808 08:24:21.485133 11050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f49b3297174224869f29b3a86f038f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260808 08:24:21.485257 11050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:21.485343 11050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:21.485462 11050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:21.486460 11050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f49b3297174224869f29b3a86f038f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260808 08:24:21.486806 11050 leader_election.cc:302] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f49b3297174224869f29b3a86f038f; no voters:
I20260808 08:24:21.487131 11050 leader_election.cc:288] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:21.487272 11066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:21.487516 11066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [term 1 LEADER]: Becoming Leader. State: Replica: 66f49b3297174224869f29b3a86f038f, State: Running, Role: LEADER
I20260808 08:24:21.488059 11066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f49b3297174224869f29b3a86f038f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260808 08:24:21.488483 11050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:21.489782 11069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66f49b3297174224869f29b3a86f038f. Latest consensus state: current_term: 1 leader_uuid: "66f49b3297174224869f29b3a86f038f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f49b3297174224869f29b3a86f038f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } } }
I20260808 08:24:21.489938 11069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:21.490345 11078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:21.490320 11068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66f49b3297174224869f29b3a86f038f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f49b3297174224869f29b3a86f038f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } } }
I20260808 08:24:21.490557 11068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:21.493176 11078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:21.498612 11078 catalog_manager.cc:1347] Generated new cluster ID: b389d2b5c63d49b69e3f56a207123443
I20260808 08:24:21.498683 11078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:21.510970 11078 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:21.511328 11078 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:21.517921 11078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66f49b3297174224869f29b3a86f038f: Generated new TSK 0
I20260808 08:24:21.518694 11078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:21.526818 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 10961
2026-08-08T08:24:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:21 dist-test-slave-6ntk krb5kdc[11091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:21 dist-test-slave-6ntk krb5kdc[11091](info): set up 2 sockets
Aug 08 08:24:21 dist-test-slave-6ntk krb5kdc[11091](info): commencing operation
krb5kdc: starting...
W20260808 08:24:22.849766 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:22 dist-test-slave-6ntk krb5kdc[11091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:22.991511 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:23.123070 11109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:23.123471 11109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:23.123585 11109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:23.128010 11109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:23.128129 11109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:23.128163 11109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:23.128203 11109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:23.128242 11109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:23.133970 11109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:23.135206 11109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:23.136392 11109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:24:23.145730 11109 server_base.cc:1034] running on GCE node
W20260808 08:24:23.145709 11122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:23.145654 11125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:23.145679 11120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:23.146762 11109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:23.147991 11109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:23.149185 11109 hybrid_clock.cc:648] HybridClock initialized: now 1786177463149164 us; error 34 us; skew 500 ppm
Aug 08 08:24:23 dist-test-slave-6ntk krb5kdc[11091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:23.152248 11109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:23.153175 11109 webserver.cc:460] Webserver started at http://127.0.0.1:44733/ using document root <none> and password file <none>
I20260808 08:24:23.153738 11109 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:23.153798 11109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:23.154057 11109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:23.156157 11109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "46343067c3324c43a83d0a2ff350add2"
format_stamp: "Formatted at 2026-08-08 08:24:23 on dist-test-slave-6ntk"
server_key: "4612be5762bbbfd85318a3d1fe7f1df1"
server_key_iv: "d32eb31282fb497fa12baeb741abc97b"
server_key_version: "encryptionkey@0"
I20260808 08:24:23.156685 11109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "46343067c3324c43a83d0a2ff350add2"
format_stamp: "Formatted at 2026-08-08 08:24:23 on dist-test-slave-6ntk"
server_key: "4612be5762bbbfd85318a3d1fe7f1df1"
server_key_iv: "d32eb31282fb497fa12baeb741abc97b"
server_key_version: "encryptionkey@0"
I20260808 08:24:23.160597 11109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 08:24:23.163218 11132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:23.164563 11109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:24:23.164713 11109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "46343067c3324c43a83d0a2ff350add2"
format_stamp: "Formatted at 2026-08-08 08:24:23 on dist-test-slave-6ntk"
server_key: "4612be5762bbbfd85318a3d1fe7f1df1"
server_key_iv: "d32eb31282fb497fa12baeb741abc97b"
server_key_version: "encryptionkey@0"
I20260808 08:24:23.164824 11109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:23.188162 11109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:23.192864 11109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:23.193084 11109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:23.205754 11109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35625
I20260808 08:24:23.205785 11198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35625 every 8 connection(s)
I20260808 08:24:23.207185 11109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:23.210289 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 11109
I20260808 08:24:23.210505 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:23.210969 12340 external_mini_cluster.cc:1428] Setting key 6c38947d489195f2793289fbd45537db
I20260808 08:24:23.211617 11199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:23.221011 11199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2: Bootstrap starting.
I20260808 08:24:23.224017 11199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2: Neither blocks nor log segments found. Creating new log.
Aug 08 08:24:23 dist-test-slave-6ntk krb5kdc[11091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 08:24:23.225669 11199 log.cc:824] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:23.229624 11199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2: No bootstrap required, opened a new log
I20260808 08:24:23.232389 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:23.213256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:58987 (local address 127.0.0.1:35625)
0808 08:24:23.213773 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:23.213789 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:23.213828 (+ 39us) server_negotiation.cc:407] Connection header received
0808 08:24:23.215095 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:23.215124 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:23.215664 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:23.216074 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:23.216988 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:23.218690 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:23.220027 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:23.221032 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:23.221981 (+ 949us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:23.225936 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:23.225946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:23.225970 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:23.226022 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:23.228566 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:23.229393 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:23.229401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:23.229410 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:23.229598 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:23.230155 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:23.230159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:23.230161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:23.230643 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:23.230904 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:23.231257 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 08:24:23.231513 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":161,"threads_started":1}
I20260808 08:24:23.233875 11199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46343067c3324c43a83d0a2ff350add2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260808 08:24:23.234186 11199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:23.234256 11199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46343067c3324c43a83d0a2ff350add2, State: Initialized, Role: FOLLOWER
I20260808 08:24:23.234728 11199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46343067c3324c43a83d0a2ff350add2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260808 08:24:23.234886 11199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:23.234956 11199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:23.235080 11199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:23.235725 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:23.236246 11199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46343067c3324c43a83d0a2ff350add2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260808 08:24:23.236608 11199 leader_election.cc:302] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46343067c3324c43a83d0a2ff350add2; no voters:
I20260808 08:24:23.236941 11199 leader_election.cc:288] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:23.237016 11208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:23.237237 11208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [term 1 LEADER]: Becoming Leader. State: Replica: 46343067c3324c43a83d0a2ff350add2, State: Running, Role: LEADER
I20260808 08:24:23.237576 11208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46343067c3324c43a83d0a2ff350add2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260808 08:24:23.238463 11199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:23.238868 11208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46343067c3324c43a83d0a2ff350add2. Latest consensus state: current_term: 1 leader_uuid: "46343067c3324c43a83d0a2ff350add2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46343067c3324c43a83d0a2ff350add2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } } }
I20260808 08:24:23.238978 11208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:23.239321 11212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46343067c3324c43a83d0a2ff350add2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46343067c3324c43a83d0a2ff350add2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } } }
I20260808 08:24:23.239434 11212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2 [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:23.243781 11232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:23.243880 11232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:23.243947 11216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:23.244889 11216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:23.247269 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:23.238030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:33071 (local address 127.0.0.1:35625)
0808 08:24:23.238146 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:23.238150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:23.238449 (+ 299us) server_negotiation.cc:407] Connection header received
0808 08:24:23.238610 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:23.238613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:23.238681 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:23.238758 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:23.239809 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:23.240604 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:23.241922 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:23.242168 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:23.242281 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:23.243928 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:23.243932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:23.243934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:23.243963 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:23.245462 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:23.245969 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:23.245972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:23.245974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:23.246061 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:23.246391 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:23.246394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:23.246396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:23.246581 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:23.246733 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:23.246998 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 08:24:23.247100 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 08:24:23.251292 11216 catalog_manager.cc:1347] Generated new cluster ID: d78e0a2aa91a4fa38aea93d27bfcf83e
I20260808 08:24:23.251364 11216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:23.263361 11216 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:23.264343 11216 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:23.272621 11216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46343067c3324c43a83d0a2ff350add2: Generated new TSK 0
I20260808 08:24:23.273360 11216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:23.278271 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 11109
W20260808 08:24:23.298527 11089 connection.cc:535] client connection to 127.0.0.1:35625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35625): Connection reset by peer (error 104)
2026-08-08T08:24:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:23 dist-test-slave-6ntk krb5kdc[11242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:23 dist-test-slave-6ntk krb5kdc[11242](info): set up 2 sockets
Aug 08 08:24:23 dist-test-slave-6ntk krb5kdc[11242](info): commencing operation
krb5kdc: starting...
W20260808 08:24:24.607964 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:24 dist-test-slave-6ntk krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:24.753367 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:24.876334 11264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:24.876631 11264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:24.876675 11264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:24.880897 11264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:24.880975 11264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:24.880996 11264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:24.881014 11264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:24.881029 11264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:24.886710 11264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:24.887889 11264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:24.889010 11264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:24.898377 11274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:24.898403 11270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:24.898377 11269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:24.898622 11264 server_base.cc:1034] running on GCE node
I20260808 08:24:24.899381 11264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:24.900604 11264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:24.901742 11264 hybrid_clock.cc:648] HybridClock initialized: now 1786177464901724 us; error 34 us; skew 500 ppm
Aug 08 08:24:24 dist-test-slave-6ntk krb5kdc[11242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:24.904820 11264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:24.905861 11264 webserver.cc:460] Webserver started at http://127.0.0.1:40725/ using document root <none> and password file <none>
I20260808 08:24:24.906563 11264 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:24.906631 11264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:24.906901 11264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:24.909271 11264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "b2116e596cc249e5adba192f41f1f995"
format_stamp: "Formatted at 2026-08-08 08:24:24 on dist-test-slave-6ntk"
server_key: "68372ea66c7dff93cf66aeb12535a5bf"
server_key_iv: "27cd47cb0cc2f997579f1b2b6ffcf445"
server_key_version: "encryptionkey@0"
I20260808 08:24:24.909965 11264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "b2116e596cc249e5adba192f41f1f995"
format_stamp: "Formatted at 2026-08-08 08:24:24 on dist-test-slave-6ntk"
server_key: "68372ea66c7dff93cf66aeb12535a5bf"
server_key_iv: "27cd47cb0cc2f997579f1b2b6ffcf445"
server_key_version: "encryptionkey@0"
I20260808 08:24:24.914165 11264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:24.916985 11285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:24.918552 11264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:24.918711 11264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "b2116e596cc249e5adba192f41f1f995"
format_stamp: "Formatted at 2026-08-08 08:24:24 on dist-test-slave-6ntk"
server_key: "68372ea66c7dff93cf66aeb12535a5bf"
server_key_iv: "27cd47cb0cc2f997579f1b2b6ffcf445"
server_key_version: "encryptionkey@0"
I20260808 08:24:24.918834 11264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:24.932580 11264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:24.935936 11264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:24.936162 11264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:24.943702 11264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41865
I20260808 08:24:24.943737 11361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41865 every 8 connection(s)
I20260808 08:24:24.944743 11264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:24.947827 11362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:24.952318 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 11264
I20260808 08:24:24.952509 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:24.952798 12340 external_mini_cluster.cc:1428] Setting key 421d048c4657d5b9e54c849b0f1f8f95
I20260808 08:24:24.954299 11362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995: Bootstrap starting.
I20260808 08:24:24.957290 11362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:24.958266 11362 log.cc:824] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:24.960482 11362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995: No bootstrap required, opened a new log
I20260808 08:24:24.963217 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:24.954417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:48117 (local address 127.0.0.1:41865)
0808 08:24:24.954911 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:24.954932 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:24.954977 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:24:24.955895 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:24.955921 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:24.956385 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:24.956774 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:24.957254 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:24.958838 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:24.959776 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:24.960269 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:24.960460 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:24.961408 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:24.961413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:24.961427 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:24.961843 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:24.961954 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:24.961982 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:24.962124 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:24.962308 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 08:24:24.962462 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260808 08:24:24.964504 11362 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2116e596cc249e5adba192f41f1f995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41865 } }
I20260808 08:24:24.964794 11362 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:24.964860 11362 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2116e596cc249e5adba192f41f1f995, State: Initialized, Role: FOLLOWER
I20260808 08:24:24.965365 11362 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2116e596cc249e5adba192f41f1f995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41865 } }
I20260808 08:24:24.965517 11362 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:24.965607 11362 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:24.965684 11362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:24.966486 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:24.966879 11362 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2116e596cc249e5adba192f41f1f995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41865 } }
I20260808 08:24:24.967238 11362 leader_election.cc:302] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2116e596cc249e5adba192f41f1f995; no voters:
I20260808 08:24:24.967636 11362 leader_election.cc:288] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:24.967718 11368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:24.967969 11368 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [term 1 LEADER]: Becoming Leader. State: Replica: b2116e596cc249e5adba192f41f1f995, State: Running, Role: LEADER
I20260808 08:24:24.968299 11368 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2116e596cc249e5adba192f41f1f995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41865 } }
I20260808 08:24:24.969094 11362 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:24.970112 11373 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2116e596cc249e5adba192f41f1f995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2116e596cc249e5adba192f41f1f995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41865 } } }
I20260808 08:24:24.970269 11373 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:24.970566 11377 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2116e596cc249e5adba192f41f1f995. Latest consensus state: current_term: 1 leader_uuid: "b2116e596cc249e5adba192f41f1f995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2116e596cc249e5adba192f41f1f995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41865 } } }
I20260808 08:24:24.970641 11377 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:24.970901 11388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:24.974375 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:24.968731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:58971 (local address 127.0.0.1:41865)
0808 08:24:24.968858 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:24.968862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:24.969273 (+ 411us) server_negotiation.cc:407] Connection header received
0808 08:24:24.969418 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:24.969421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:24.969474 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:24.969580 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:24.970130 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:24.970953 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:24.972309 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:24.972722 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:24.972836 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:24.973804 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:24.973808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:24.973812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:24.973829 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:24.973857 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:24.973893 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:24.973987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:24.974190 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 08:24:24.974228 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 08:24:24.974515 11388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:24.980631 11388 catalog_manager.cc:1347] Generated new cluster ID: 53c2aed70fe24b79a3d8302468295f4b
I20260808 08:24:24.980723 11388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:24.995098 11388 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:24.996415 11388 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:25.009564 11388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2116e596cc249e5adba192f41f1f995: Generated new TSK 0
I20260808 08:24:25.010599 11388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:25.020727 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 11264
W20260808 08:24:25.030577 11238 connection.cc:535] client connection to 127.0.0.1:41865 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41865): Connection reset by peer (error 104)
2026-08-08T08:24:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1731 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:25 dist-test-slave-6ntk krb5kdc[11404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:25 dist-test-slave-6ntk krb5kdc[11404](info): set up 2 sockets
Aug 08 08:24:25 dist-test-slave-6ntk krb5kdc[11404](info): commencing operation
krb5kdc: starting...
W20260808 08:24:26.345553 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:26 dist-test-slave-6ntk krb5kdc[11404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:26.504887 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:26.626058 11425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:26.626420 11425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:26.626500 11425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:26.630642 11425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:26.630755 11425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:26.630795 11425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:26.630833 11425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:26.630864 11425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:26.636446 11425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:26.637813 11425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:26.638985 11425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:26.648512 11432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:26.648504 11431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:26.648504 11435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:26.648783 11425 server_base.cc:1034] running on GCE node
I20260808 08:24:26.649519 11425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:26.650800 11425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:26.652006 11425 hybrid_clock.cc:648] HybridClock initialized: now 1786177466651992 us; error 31 us; skew 500 ppm
Aug 08 08:24:26 dist-test-slave-6ntk krb5kdc[11404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:26.659036 11425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:26.660163 11425 webserver.cc:460] Webserver started at http://127.0.0.1:44643/ using document root <none> and password file <none>
I20260808 08:24:26.660796 11425 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:26.660892 11425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:26.661154 11425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:26.663447 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "060add33974c4838ae8710da81c78a17"
format_stamp: "Formatted at 2026-08-08 08:24:26 on dist-test-slave-6ntk"
server_key: "97a65bbeb63936b3c655ed272c40ef77"
server_key_iv: "565393faf119ec33c72a7319f55068e3"
server_key_version: "encryptionkey@0"
I20260808 08:24:26.664069 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "060add33974c4838ae8710da81c78a17"
format_stamp: "Formatted at 2026-08-08 08:24:26 on dist-test-slave-6ntk"
server_key: "97a65bbeb63936b3c655ed272c40ef77"
server_key_iv: "565393faf119ec33c72a7319f55068e3"
server_key_version: "encryptionkey@0"
I20260808 08:24:26.668175 11425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:26.670820 11444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:26.672199 11425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:26.672371 11425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "060add33974c4838ae8710da81c78a17"
format_stamp: "Formatted at 2026-08-08 08:24:26 on dist-test-slave-6ntk"
server_key: "97a65bbeb63936b3c655ed272c40ef77"
server_key_iv: "565393faf119ec33c72a7319f55068e3"
server_key_version: "encryptionkey@0"
I20260808 08:24:26.672497 11425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:26.677731 11425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:26.681066 11425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:26.681380 11425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:26.689458 11425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35587
I20260808 08:24:26.689488 11511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35587 every 8 connection(s)
I20260808 08:24:26.690546 11425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:26.693470 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:26.693910 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 11425
I20260808 08:24:26.694067 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:26.694372 12340 external_mini_cluster.cc:1428] Setting key bd8c71949c131c99ec7fc70d066ac55d
I20260808 08:24:26.699759 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:26.695812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:49515 (local address 127.0.0.1:35587)
0808 08:24:26.696298 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:26.696313 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:26.696384 (+ 71us) server_negotiation.cc:407] Connection header received
0808 08:24:26.697078 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:26.697097 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:26.697398 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:26.697754 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:26.698585 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:26.698590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:26.698602 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:26.698613 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:26.698668 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:26.698688 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:26.698839 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:26.698917 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 08:24:26.699082 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":137,"threads_started":1}
I20260808 08:24:26.700711 11512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17: Bootstrap starting.
I20260808 08:24:26.702622 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:26.703284 11512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:26.704183 11512 log.cc:824] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:26.706562 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:26.704834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:34451 (local address 127.0.0.1:35587)
0808 08:24:26.704978 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:26.704983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:26.705187 (+ 204us) server_negotiation.cc:407] Connection header received
0808 08:24:26.705268 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:26.705272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:26.705303 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:26.705379 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:26.706161 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:26.706165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:26.706167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:26.706180 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:26.706209 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:26.706222 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:26.706321 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:26.706400 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 08:24:26.706444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 08:24:26.706750 11512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17: No bootstrap required, opened a new log
I20260808 08:24:26.710666 11512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "060add33974c4838ae8710da81c78a17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260808 08:24:26.710906 11512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:26.710983 11512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060add33974c4838ae8710da81c78a17, State: Initialized, Role: FOLLOWER
I20260808 08:24:26.711427 11512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "060add33974c4838ae8710da81c78a17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260808 08:24:26.711556 11512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:26.711625 11512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:26.711748 11512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:26.713117 11512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "060add33974c4838ae8710da81c78a17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260808 08:24:26.713541 11512 leader_election.cc:302] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 060add33974c4838ae8710da81c78a17; no voters:
I20260808 08:24:26.713939 11512 leader_election.cc:288] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:26.714025 11526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:26.714306 11526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [term 1 LEADER]: Becoming Leader. State: Replica: 060add33974c4838ae8710da81c78a17, State: Running, Role: LEADER
I20260808 08:24:26.714684 11526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "060add33974c4838ae8710da81c78a17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260808 08:24:26.715289 11512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:26.716463 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 060add33974c4838ae8710da81c78a17. Latest consensus state: current_term: 1 leader_uuid: "060add33974c4838ae8710da81c78a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "060add33974c4838ae8710da81c78a17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260808 08:24:26.716490 11528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "060add33974c4838ae8710da81c78a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "060add33974c4838ae8710da81c78a17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260808 08:24:26.716600 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:26.716609 11528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:26.717115 11541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:26.720124 11541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:26.725415 11541 catalog_manager.cc:1347] Generated new cluster ID: df449576859d4614867cef777c06b533
I20260808 08:24:26.725490 11541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:26.763764 11541 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:26.764139 11541 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:26.772842 11541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 060add33974c4838ae8710da81c78a17: Generated new TSK 0
I20260808 08:24:26.773701 11541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:26.777088 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 11425
2026-08-08T08:24:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1761 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:26 dist-test-slave-6ntk krb5kdc[11563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:26 dist-test-slave-6ntk krb5kdc[11563](info): set up 2 sockets
Aug 08 08:24:26 dist-test-slave-6ntk krb5kdc[11563](info): commencing operation
krb5kdc: starting...
W20260808 08:24:28.087620 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:28 dist-test-slave-6ntk krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:28.231202 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:28.351373 11584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:28.351673 11584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:28.351716 11584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:28.355818 11584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:28.355888 11584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:28.355908 11584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:28.355924 11584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:28.355978 11584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:28.361538 11584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:28.362766 11584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:28.363880 11584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:28.373308 11592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:28.373308 11593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:28.373447 11595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:28.373716 11584 server_base.cc:1034] running on GCE node
I20260808 08:24:28.374414 11584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:28.375648 11584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:28.376859 11584 hybrid_clock.cc:648] HybridClock initialized: now 1786177468376841 us; error 31 us; skew 500 ppm
Aug 08 08:24:28 dist-test-slave-6ntk krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:28.379865 11584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:28.380769 11584 webserver.cc:460] Webserver started at http://127.0.0.1:46081/ using document root <none> and password file <none>
I20260808 08:24:28.381318 11584 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:28.381387 11584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:28.381561 11584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:28.383754 11584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "ea8e668770c64178af2528ca2add983b"
format_stamp: "Formatted at 2026-08-08 08:24:28 on dist-test-slave-6ntk"
server_key: "f6a168b7663af1b97bebb255e028822e"
server_key_iv: "a2eb93bc7a19a830b12418e7fb1bdf53"
server_key_version: "encryptionkey@0"
I20260808 08:24:28.384316 11584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "ea8e668770c64178af2528ca2add983b"
format_stamp: "Formatted at 2026-08-08 08:24:28 on dist-test-slave-6ntk"
server_key: "f6a168b7663af1b97bebb255e028822e"
server_key_iv: "a2eb93bc7a19a830b12418e7fb1bdf53"
server_key_version: "encryptionkey@0"
I20260808 08:24:28.388381 11584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:28.391000 11605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:28.392361 11584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:28.392478 11584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "ea8e668770c64178af2528ca2add983b"
format_stamp: "Formatted at 2026-08-08 08:24:28 on dist-test-slave-6ntk"
server_key: "f6a168b7663af1b97bebb255e028822e"
server_key_iv: "a2eb93bc7a19a830b12418e7fb1bdf53"
server_key_version: "encryptionkey@0"
I20260808 08:24:28.392571 11584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:28.473241 11584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:28.476585 11584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:28.476800 11584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:28.484897 11584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40535
I20260808 08:24:28.484917 11680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40535 every 8 connection(s)
I20260808 08:24:28.486048 11584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:28.489146 11682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:28.490783 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 11584
I20260808 08:24:28.490947 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:28.491241 12340 external_mini_cluster.cc:1428] Setting key dc8b429d4c10db9351c1987fca02a804
I20260808 08:24:28.495733 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b: Bootstrap starting.
I20260808 08:24:28.498437 11682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:28.499251 11682 log.cc:824] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:24:28 dist-test-slave-6ntk krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 08:24:28.501562 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b: No bootstrap required, opened a new log
I20260808 08:24:28.505573 11682 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8e668770c64178af2528ca2add983b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260808 08:24:28.505842 11682 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:28.505992 11682 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8e668770c64178af2528ca2add983b, State: Initialized, Role: FOLLOWER
I20260808 08:24:28.506536 11682 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8e668770c64178af2528ca2add983b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260808 08:24:28.506700 11682 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:28.506791 11682 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:28.506912 11682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:28.507540 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:28.492802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:56753 (local address 127.0.0.1:40535)
0808 08:24:28.493280 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:28.493296 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:28.493342 (+ 46us) server_negotiation.cc:407] Connection header received
0808 08:24:28.494055 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:28.494076 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:28.494454 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:28.494807 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:28.495703 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:28.496813 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:28.498076 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:28.498744 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:28.498958 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:28.502176 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:28.502182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:28.502197 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:28.502227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:28.504100 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:28.504680 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:28.504685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:28.504691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:28.504813 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:28.505184 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:28.505194 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:28.505196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:28.505612 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:28.505880 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:28.506250 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 08:24:28.506527 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":143,"threads_started":1}
I20260808 08:24:28.508064 11682 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8e668770c64178af2528ca2add983b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260808 08:24:28.508455 11682 leader_election.cc:302] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea8e668770c64178af2528ca2add983b; no voters:
I20260808 08:24:28.508823 11682 leader_election.cc:288] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:28.508940 11690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:28.509148 11690 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [term 1 LEADER]: Becoming Leader. State: Replica: ea8e668770c64178af2528ca2add983b, State: Running, Role: LEADER
I20260808 08:24:28.509554 11690 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8e668770c64178af2528ca2add983b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260808 08:24:28.510296 11682 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:28.511183 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:28.514070 11691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea8e668770c64178af2528ca2add983b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8e668770c64178af2528ca2add983b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } } }
I20260808 08:24:28.514231 11691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:28.514617 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea8e668770c64178af2528ca2add983b. Latest consensus state: current_term: 1 leader_uuid: "ea8e668770c64178af2528ca2add983b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8e668770c64178af2528ca2add983b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } } }
I20260808 08:24:28.514719 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:28.515730 11711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:28.515851 11711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:28.515952 11722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:28.516705 11722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:28.523360 11722 catalog_manager.cc:1347] Generated new cluster ID: 4de329ac36084989abcec4b2517f5d79
I20260808 08:24:28.523443 11722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:28.527552 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:28.515093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:40465 (local address 127.0.0.1:40535)
0808 08:24:28.515219 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:28.515223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:28.515474 (+ 251us) server_negotiation.cc:407] Connection header received
0808 08:24:28.515645 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:28.515648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:28.515704 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:28.515787 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:28.519358 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:28.520146 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:28.521205 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:28.521488 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:28.521617 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:28.523626 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:28.523630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:28.523633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:28.523661 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:28.525478 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:28.526131 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:28.526135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:28.526137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:28.526227 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:28.526605 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:28.526608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:28.526610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:28.526822 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:28.526981 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:28.527290 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 08:24:28.527392 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 08:24:28.542842 11722 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:28.544282 11722 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:28.555075 11722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea8e668770c64178af2528ca2add983b: Generated new TSK 0
I20260808 08:24:28.555934 11722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:28.567411 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 11584
W20260808 08:24:28.579914 11561 connection.cc:535] client connection to 127.0.0.1:40535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40535): Connection reset by peer (error 104)
2026-08-08T08:24:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1788 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:28 dist-test-slave-6ntk krb5kdc[11728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:28 dist-test-slave-6ntk krb5kdc[11728](info): set up 2 sockets
Aug 08 08:24:28 dist-test-slave-6ntk krb5kdc[11728](info): commencing operation
krb5kdc: starting...
W20260808 08:24:29.823401 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:29 dist-test-slave-6ntk krb5kdc[11728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:29.962821 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:30.081379 11749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:30.081746 11749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:30.081840 11749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:30.085947 11749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:30.086057 11749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:30.086092 11749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:30.086131 11749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:30.086167 11749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:30.091581 11749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:30.092795 11749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:30.093982 11749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:30.103176 11755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:30.103192 11756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:30.103298 11749 server_base.cc:1034] running on GCE node
W20260808 08:24:30.103199 11759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:30.104081 11749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:30.105238 11749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:30.106410 11749 hybrid_clock.cc:648] HybridClock initialized: now 1786177470106382 us; error 41 us; skew 500 ppm
Aug 08 08:24:30 dist-test-slave-6ntk krb5kdc[11728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:30.109565 11749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:30.110597 11749 webserver.cc:460] Webserver started at http://127.0.0.1:35891/ using document root <none> and password file <none>
I20260808 08:24:30.111218 11749 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:30.111282 11749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:30.111537 11749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:30.113741 11749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "617de0f88c804d629c23ba238677d31c"
format_stamp: "Formatted at 2026-08-08 08:24:30 on dist-test-slave-6ntk"
server_key: "1b93b14dff2bb2e2a3c72b47d87f9acb"
server_key_iv: "71b6de2730eb9a37be5b592767c5bec7"
server_key_version: "encryptionkey@0"
I20260808 08:24:30.114475 11749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "617de0f88c804d629c23ba238677d31c"
format_stamp: "Formatted at 2026-08-08 08:24:30 on dist-test-slave-6ntk"
server_key: "1b93b14dff2bb2e2a3c72b47d87f9acb"
server_key_iv: "71b6de2730eb9a37be5b592767c5bec7"
server_key_version: "encryptionkey@0"
I20260808 08:24:30.118615 11749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:30.121243 11769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:30.122625 11749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:24:30.122784 11749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "617de0f88c804d629c23ba238677d31c"
format_stamp: "Formatted at 2026-08-08 08:24:30 on dist-test-slave-6ntk"
server_key: "1b93b14dff2bb2e2a3c72b47d87f9acb"
server_key_iv: "71b6de2730eb9a37be5b592767c5bec7"
server_key_version: "encryptionkey@0"
I20260808 08:24:30.122910 11749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:30.134320 11749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:30.137322 11749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:30.137549 11749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:30.145164 11749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45969
I20260808 08:24:30.145195 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45969 every 8 connection(s)
I20260808 08:24:30.146260 11749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:30.149327 11843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:30.150481 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 11749
I20260808 08:24:30.150642 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:30.151011 12340 external_mini_cluster.cc:1428] Setting key 31b99b67d50198c889ed016df255b0e1
I20260808 08:24:30.156032 11843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c: Bootstrap starting.
I20260808 08:24:30.158583 11843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:30.159363 11843 log.cc:824] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:30.160604 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:30.152564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:50367 (local address 127.0.0.1:45969)
0808 08:24:30.153037 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:30.153052 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:30.153090 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:24:30.153744 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:30.153762 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:30.154209 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:30.154563 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:30.154985 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:30.155962 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:30.156935 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:30.157423 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:30.157615 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:30.158595 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:30.158600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:30.158614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:30.159087 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:30.159154 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:30.159178 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:30.159337 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:30.159520 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 08:24:30.159672 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260808 08:24:30.161707 11843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c: No bootstrap required, opened a new log
I20260808 08:24:30.163810 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:30.166160 11843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617de0f88c804d629c23ba238677d31c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260808 08:24:30.166534 11843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:30.166615 11843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617de0f88c804d629c23ba238677d31c, State: Initialized, Role: FOLLOWER
I20260808 08:24:30.167088 11843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617de0f88c804d629c23ba238677d31c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260808 08:24:30.167249 11843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:30.167340 11843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:30.167469 11843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:30.168466 11843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617de0f88c804d629c23ba238677d31c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260808 08:24:30.168838 11843 leader_election.cc:302] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617de0f88c804d629c23ba238677d31c; no voters:
I20260808 08:24:30.169281 11843 leader_election.cc:288] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:30.169333 11858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:30.169544 11858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [term 1 LEADER]: Becoming Leader. State: Replica: 617de0f88c804d629c23ba238677d31c, State: Running, Role: LEADER
I20260808 08:24:30.169912 11858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617de0f88c804d629c23ba238677d31c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260808 08:24:30.170616 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:30.165977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:52685 (local address 127.0.0.1:45969)
0808 08:24:30.166093 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:30.166096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:30.166353 (+ 257us) server_negotiation.cc:407] Connection header received
0808 08:24:30.166542 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:30.166545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:30.166588 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:30.166686 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:30.166995 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:30.167676 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:30.168514 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:30.168832 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:30.168945 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:30.170045 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:30.170049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:30.170051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:30.170064 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:30.170091 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:30.170104 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:30.170191 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:30.170434 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 08:24:30.170472 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 08:24:30.170749 11843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:30.171967 11859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617de0f88c804d629c23ba238677d31c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617de0f88c804d629c23ba238677d31c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } } }
I20260808 08:24:30.172107 11859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:30.172314 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617de0f88c804d629c23ba238677d31c. Latest consensus state: current_term: 1 leader_uuid: "617de0f88c804d629c23ba238677d31c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617de0f88c804d629c23ba238677d31c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } } }
I20260808 08:24:30.172530 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:30.173291 11874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:30.176127 11874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:30.181771 11874 catalog_manager.cc:1347] Generated new cluster ID: 16f473ee397e49e2b551a2f041e5b94f
I20260808 08:24:30.181850 11874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:30.195667 11874 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:30.197098 11874 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:30.206936 11874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617de0f88c804d629c23ba238677d31c: Generated new TSK 0
I20260808 08:24:30.207819 11874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:30.218393 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 11749
W20260808 08:24:30.230989 11726 connection.cc:535] client connection to 127.0.0.1:45969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45969): Connection reset by peer (error 104)
2026-08-08T08:24:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1651 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:30 dist-test-slave-6ntk krb5kdc[11893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:30 dist-test-slave-6ntk krb5kdc[11893](info): set up 2 sockets
Aug 08 08:24:30 dist-test-slave-6ntk krb5kdc[11893](info): commencing operation
krb5kdc: starting...
W20260808 08:24:31.470887 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:31 dist-test-slave-6ntk krb5kdc[11893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 08:24:31.610100 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:31.731943 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:31.732309 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:31.732398 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:31.736517 11912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:31.736624 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:31.736663 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:31.736706 11912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:31.736774 11912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:31.742273 11912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:31.743465 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:31.744611 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:31.753733 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:31.753834 11919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:31.753733 11918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:31.754139 11912 server_base.cc:1034] running on GCE node
I20260808 08:24:31.754784 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:31.756003 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:31.757280 11912 hybrid_clock.cc:648] HybridClock initialized: now 1786177471757220 us; error 45 us; skew 500 ppm
Aug 08 08:24:31 dist-test-slave-6ntk krb5kdc[11893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:31.766151 11912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 08:24:31.767156 11912 webserver.cc:460] Webserver started at http://127.0.0.1:46741/ using document root <none> and password file <none>
I20260808 08:24:31.767875 11912 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:31.767974 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:31.768231 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:31.770524 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "d87d0f37dfa3445eb5cc7164f03141ae"
format_stamp: "Formatted at 2026-08-08 08:24:31 on dist-test-slave-6ntk"
server_key: "f0cd6e8c96802039eaffc0cf2729feda"
server_key_iv: "7e5bcaeaaf2fefb9dc2888641af1009b"
server_key_version: "encryptionkey@0"
I20260808 08:24:31.771107 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "d87d0f37dfa3445eb5cc7164f03141ae"
format_stamp: "Formatted at 2026-08-08 08:24:31 on dist-test-slave-6ntk"
server_key: "f0cd6e8c96802039eaffc0cf2729feda"
server_key_iv: "7e5bcaeaaf2fefb9dc2888641af1009b"
server_key_version: "encryptionkey@0"
I20260808 08:24:31.775039 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 08:24:31.777621 11933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:31.778967 11912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 08:24:31.779074 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "d87d0f37dfa3445eb5cc7164f03141ae"
format_stamp: "Formatted at 2026-08-08 08:24:31 on dist-test-slave-6ntk"
server_key: "f0cd6e8c96802039eaffc0cf2729feda"
server_key_iv: "7e5bcaeaaf2fefb9dc2888641af1009b"
server_key_version: "encryptionkey@0"
I20260808 08:24:31.779161 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:31.783794 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:31.786777 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:31.786923 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:31.794157 11912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38429
I20260808 08:24:31.794185 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38429 every 8 connection(s)
I20260808 08:24:31.795121 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:31.798110 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:31.798177 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 11912
I20260808 08:24:31.798362 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:31.798673 12340 external_mini_cluster.cc:1428] Setting key dae744a6bcaa0a13c0d5eae50d03d4f0
I20260808 08:24:31.804176 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:31.800162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:51773 (local address 127.0.0.1:38429)
0808 08:24:31.800680 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:31.800694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:31.800728 (+ 34us) server_negotiation.cc:407] Connection header received
0808 08:24:31.801459 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:31.801478 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:31.801760 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:31.802169 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:31.803003 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:31.803007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:31.803019 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:31.803031 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:31.803087 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:31.803106 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:31.803261 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:31.803343 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 08:24:31.803507 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":128,"threads_started":1}
I20260808 08:24:31.805091 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae: Bootstrap starting.
I20260808 08:24:31.807019 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:31.807663 12004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:31.808499 12004 log.cc:824] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:31.810876 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae: No bootstrap required, opened a new log
I20260808 08:24:31.811583 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:31.809169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:58903 (local address 127.0.0.1:38429)
0808 08:24:31.809309 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:31.809314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:31.810165 (+ 851us) server_negotiation.cc:407] Connection header received
0808 08:24:31.810288 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:31.810290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:31.810321 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:31.810400 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:31.811143 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:31.811146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:31.811148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:31.811159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:31.811185 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:31.811195 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:31.811268 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:31.811400 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 08:24:31.811435 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 08:24:31.814827 12004 raft_consensus.cc:348] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 08:24:31.815099 12004 raft_consensus.cc:374] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:31.815179 12004 raft_consensus.cc:729] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87d0f37dfa3445eb5cc7164f03141ae, State: Initialized, Role: FOLLOWER
I20260808 08:24:31.815623 12004 consensus_queue.cc:260] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 08:24:31.815784 12004 raft_consensus.cc:388] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:31.815874 12004 raft_consensus.cc:482] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:31.815980 12004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:31.816970 12004 raft_consensus.cc:504] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 08:24:31.817348 12004 leader_election.cc:302] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d87d0f37dfa3445eb5cc7164f03141ae; no voters:
I20260808 08:24:31.817698 12004 leader_election.cc:288] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:31.817934 12017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:31.818301 12017 raft_consensus.cc:686] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [term 1 LEADER]: Becoming Leader. State: Replica: d87d0f37dfa3445eb5cc7164f03141ae, State: Running, Role: LEADER
I20260808 08:24:31.818974 12017 consensus_queue.cc:237] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 08:24:31.819175 12004 sys_catalog.cc:564] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:31.820775 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } } }
I20260808 08:24:31.820914 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:31.821262 12019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d87d0f37dfa3445eb5cc7164f03141ae. Latest consensus state: current_term: 1 leader_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87d0f37dfa3445eb5cc7164f03141ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } } }
I20260808 08:24:31.821774 12019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:31.824311 12037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:31.824380 12037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:31.824466 12036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:31.825196 12036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:31.831128 12036 catalog_manager.cc:1347] Generated new cluster ID: 10a68af990e84a80b8887ed74ff54720
I20260808 08:24:31.831205 12036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:31.851207 12036 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:31.851589 12036 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:31.861169 12036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d87d0f37dfa3445eb5cc7164f03141ae: Generated new TSK 0
I20260808 08:24:31.862077 12036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:31.866070 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 11912
2026-08-08T08:24:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1650 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:31 dist-test-slave-6ntk krb5kdc[12049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:31 dist-test-slave-6ntk krb5kdc[12049](info): set up 2 sockets
Aug 08 08:24:31 dist-test-slave-6ntk krb5kdc[12049](info): commencing operation
krb5kdc: starting...
W20260808 08:24:33.146968 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:33 dist-test-slave-6ntk krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:33.287613 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36995
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:33.407109 12064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:33.407387 12064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:33.407429 12064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:33.411684 12064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:33.411768 12064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:33.411801 12064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:33.411842 12064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:33.411866 12064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:33.417413 12064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:33.418655 12064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:33.419821 12064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:33.429042 12076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:33.429057 12073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:33.429098 12064 server_base.cc:1034] running on GCE node
W20260808 08:24:33.429045 12074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:33.430051 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:33.431245 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:33.432443 12064 hybrid_clock.cc:648] HybridClock initialized: now 1786177473432412 us; error 46 us; skew 500 ppm
Aug 08 08:24:33 dist-test-slave-6ntk krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:33.435597 12064 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:33.436508 12064 webserver.cc:460] Webserver started at http://127.12.13.62:45161/ using document root <none> and password file <none>
I20260808 08:24:33.437071 12064 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:33.437130 12064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:33.437304 12064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:33.439478 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "9a7de61cdd3b44288c24b53fa568dc09"
format_stamp: "Formatted at 2026-08-08 08:24:33 on dist-test-slave-6ntk"
server_key: "f625346edfb39c72a25a941b7d815bf6"
server_key_iv: "86984aab241cf9029c023204e980705f"
server_key_version: "encryptionkey@0"
I20260808 08:24:33.440033 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "9a7de61cdd3b44288c24b53fa568dc09"
format_stamp: "Formatted at 2026-08-08 08:24:33 on dist-test-slave-6ntk"
server_key: "f625346edfb39c72a25a941b7d815bf6"
server_key_iv: "86984aab241cf9029c023204e980705f"
server_key_version: "encryptionkey@0"
I20260808 08:24:33.444058 12064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 08:24:33.446707 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:33.448015 12064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:33.448159 12064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "9a7de61cdd3b44288c24b53fa568dc09"
format_stamp: "Formatted at 2026-08-08 08:24:33 on dist-test-slave-6ntk"
server_key: "f625346edfb39c72a25a941b7d815bf6"
server_key_iv: "86984aab241cf9029c023204e980705f"
server_key_version: "encryptionkey@0"
I20260808 08:24:33.448272 12064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:33.484903 12064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:33.488668 12064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:33.488900 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:33.497570 12064 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36995
I20260808 08:24:33.497601 12162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36995 every 8 connection(s)
I20260808 08:24:33.498646 12064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:33.501538 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:33.506039 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12064
I20260808 08:24:33.506186 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:33.506482 12340 external_mini_cluster.cc:1428] Setting key dc0f1e44f599b6588870be3157ab71dc
I20260808 08:24:33.507927 12163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09: Bootstrap starting.
I20260808 08:24:33.510545 12163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:33.511459 12163 log.cc:824] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:33.513849 12163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09: No bootstrap required, opened a new log
Aug 08 08:24:33 dist-test-slave-6ntk krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:33.518553 12163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7de61cdd3b44288c24b53fa568dc09" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36995 } }
I20260808 08:24:33.518837 12163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:33.518930 12163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7de61cdd3b44288c24b53fa568dc09, State: Initialized, Role: FOLLOWER
I20260808 08:24:33.519472 12163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7de61cdd3b44288c24b53fa568dc09" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36995 } }
I20260808 08:24:33.519641 12163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:33.519733 12163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:33.519845 12163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:33.520843 12163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7de61cdd3b44288c24b53fa568dc09" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36995 } }
I20260808 08:24:33.521243 12163 leader_election.cc:302] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a7de61cdd3b44288c24b53fa568dc09; no voters:
I20260808 08:24:33.521641 12163 leader_election.cc:288] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:33.521757 12170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:33.522017 12170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [term 1 LEADER]: Becoming Leader. State: Replica: 9a7de61cdd3b44288c24b53fa568dc09, State: Running, Role: LEADER
I20260808 08:24:33.522374 12170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7de61cdd3b44288c24b53fa568dc09" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36995 } }
I20260808 08:24:33.523181 12163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:33.523566 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:33.507996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41889 (local address 127.12.13.62:36995)
0808 08:24:33.508496 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:33.508513 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:33.508550 (+ 37us) server_negotiation.cc:407] Connection header received
0808 08:24:33.509319 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:33.509339 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:33.509702 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:33.510080 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:33.511112 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:33.512503 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:33.513682 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:33.514329 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:33.514458 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:33.517563 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:33.517568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:33.517583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:33.517613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:33.519709 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:33.520311 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:33.520318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:33.520324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:33.520458 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:33.520844 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:33.520847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:33.520849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:33.521344 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:33.521586 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:33.522323 (+ 737us) server_negotiation.cc:299] Negotiation successful
0808 08:24:33.522599 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":144,"threads_started":1}
I20260808 08:24:33.523729 12170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a7de61cdd3b44288c24b53fa568dc09. Latest consensus state: current_term: 1 leader_uuid: "9a7de61cdd3b44288c24b53fa568dc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7de61cdd3b44288c24b53fa568dc09" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36995 } } }
I20260808 08:24:33.524214 12172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a7de61cdd3b44288c24b53fa568dc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7de61cdd3b44288c24b53fa568dc09" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36995 } } }
I20260808 08:24:33.526090 12170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:33.526091 12172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:33.527798 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 08:24:33.528647 12189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:33.528733 12189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:33.528795 12190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:33.529698 12190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:33.535821 12190 catalog_manager.cc:1347] Generated new cluster ID: d58c8e20926e4d429a943de404d652af
I20260808 08:24:33.535902 12190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:33.545421 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:33.534313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56981 (local address 127.12.13.62:36995)
0808 08:24:33.534454 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:33.534459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:33.534873 (+ 414us) server_negotiation.cc:407] Connection header received
0808 08:24:33.535023 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:33.535026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:33.535081 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:33.535176 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:33.538441 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:33.538983 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:33.539869 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:33.540112 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:33.540237 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:33.541795 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:33.541799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:33.541801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:33.541826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:33.543488 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:33.544016 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:33.544020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:33.544022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:33.544113 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:33.544464 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:33.544467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:33.544469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:33.544665 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:33.544828 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:33.545124 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 08:24:33.545229 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:24:33.573388 12190 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:33.574774 12190 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:33.584549 12190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a7de61cdd3b44288c24b53fa568dc09: Generated new TSK 0
I20260808 08:24:33.585302 12190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:33.593771 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12064
W20260808 08:24:33.609365 12046 connection.cc:535] client connection to 127.12.13.62:36995 recv error: Network error: failed to read from TLS socket (remote: 127.12.13.62:36995): Connection reset by peer (error 104)
2026-08-08T08:24:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1727 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:33 dist-test-slave-6ntk krb5kdc[12204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:33 dist-test-slave-6ntk krb5kdc[12204](info): set up 2 sockets
Aug 08 08:24:33 dist-test-slave-6ntk krb5kdc[12204](info): commencing operation
krb5kdc: starting...
W20260808 08:24:34.886379 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:34 dist-test-slave-6ntk krb5kdc[12204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:35.031208 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35641
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:35.157094 12221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:35.157478 12221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:35.157526 12221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:35.161791 12221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:35.161931 12221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:35.161971 12221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:35.161990 12221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:35.162042 12221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:35.167725 12221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:35641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:35.168924 12221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:35.170152 12221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:35.179286 12235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:35.179286 12230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:35.179589 12221 server_base.cc:1034] running on GCE node
W20260808 08:24:35.179329 12233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:35.180435 12221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:35.182117 12221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:35.183424 12221 hybrid_clock.cc:648] HybridClock initialized: now 1786177475183397 us; error 64 us; skew 500 ppm
Aug 08 08:24:35 dist-test-slave-6ntk krb5kdc[12204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177475, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:35.186971 12221 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:35.188011 12221 webserver.cc:460] Webserver started at http://127.12.13.62:34147/ using document root <none> and password file <none>
I20260808 08:24:35.188778 12221 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:35.188877 12221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:35.189128 12221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:35.191860 12221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "a3f454d712f8459a92976aaa659fd81e"
format_stamp: "Formatted at 2026-08-08 08:24:35 on dist-test-slave-6ntk"
server_key: "0b6f2d6edc80204784110120239cef4c"
server_key_iv: "298caa9854ad8102d4389b54e12ca840"
server_key_version: "encryptionkey@0"
I20260808 08:24:35.192569 12221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "a3f454d712f8459a92976aaa659fd81e"
format_stamp: "Formatted at 2026-08-08 08:24:35 on dist-test-slave-6ntk"
server_key: "0b6f2d6edc80204784110120239cef4c"
server_key_iv: "298caa9854ad8102d4389b54e12ca840"
server_key_version: "encryptionkey@0"
I20260808 08:24:35.197117 12221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:35.200145 12245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:35.201553 12221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 08:24:35.201730 12221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "a3f454d712f8459a92976aaa659fd81e"
format_stamp: "Formatted at 2026-08-08 08:24:35 on dist-test-slave-6ntk"
server_key: "0b6f2d6edc80204784110120239cef4c"
server_key_iv: "298caa9854ad8102d4389b54e12ca840"
server_key_version: "encryptionkey@0"
I20260808 08:24:35.201874 12221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:35.218320 12221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:35.221776 12221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:35.222122 12221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:35.230271 12221 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:35641
I20260808 08:24:35.230290 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:35641 every 8 connection(s)
I20260808 08:24:35.231390 12221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:35.234442 12322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:35.239696 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12221
I20260808 08:24:35.239898 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:35.240185 12340 external_mini_cluster.cc:1428] Setting key 21450744f6aa0a6dae3b2b0a09b6c566
I20260808 08:24:35.240664 12322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e: Bootstrap starting.
I20260808 08:24:35.243635 12322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:35.244598 12322 log.cc:824] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:35.246958 12322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e: No bootstrap required, opened a new log
I20260808 08:24:35.251176 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:35.241749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42325 (local address 127.12.13.62:35641)
0808 08:24:35.242268 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:35.242295 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:35.242353 (+ 58us) server_negotiation.cc:407] Connection header received
0808 08:24:35.243209 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:35.243234 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:35.243715 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:35.244125 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:35.244590 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:35.246030 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:35.246992 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:35.247709 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:35.247875 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:35.249194 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:35.249200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:35.249214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:35.249632 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:35.249696 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:35.249721 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:35.249833 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:35.250134 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 08:24:35.250349 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":182,"threads_started":1}
I20260808 08:24:35.251547 12322 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f454d712f8459a92976aaa659fd81e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35641 } }
I20260808 08:24:35.251817 12322 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:35.251910 12322 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f454d712f8459a92976aaa659fd81e, State: Initialized, Role: FOLLOWER
I20260808 08:24:35.252478 12322 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f454d712f8459a92976aaa659fd81e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35641 } }
I20260808 08:24:35.252647 12322 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:35.252715 12322 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:35.252811 12322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:35.254112 12322 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f454d712f8459a92976aaa659fd81e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35641 } }
I20260808 08:24:35.254498 12322 leader_election.cc:302] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f454d712f8459a92976aaa659fd81e; no voters:
I20260808 08:24:35.254698 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:35.254905 12322 leader_election.cc:288] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:35.254993 12328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:35.255239 12328 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [term 1 LEADER]: Becoming Leader. State: Replica: a3f454d712f8459a92976aaa659fd81e, State: Running, Role: LEADER
I20260808 08:24:35.255596 12328 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f454d712f8459a92976aaa659fd81e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35641 } }
I20260808 08:24:35.256381 12322 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:35.257099 12328 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f454d712f8459a92976aaa659fd81e. Latest consensus state: current_term: 1 leader_uuid: "a3f454d712f8459a92976aaa659fd81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f454d712f8459a92976aaa659fd81e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35641 } } }
I20260808 08:24:35.257243 12328 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:35.257741 12329 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f454d712f8459a92976aaa659fd81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f454d712f8459a92976aaa659fd81e" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35641 } } }
I20260808 08:24:35.257851 12329 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:35.262617 12364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:35.262706 12364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:35.262807 12351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:35.263850 12351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:35.263967 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:35.257184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36615 (local address 127.12.13.62:35641)
0808 08:24:35.257299 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:35.257304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:35.259328 (+ 2024us) server_negotiation.cc:407] Connection header received
0808 08:24:35.259548 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:35.259552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:35.259601 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:35.259698 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:35.260118 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:35.260642 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:35.261787 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:35.262206 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:35.262318 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:35.263353 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:35.263357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:35.263359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:35.263374 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:35.263406 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:35.263420 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:35.263524 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:35.263725 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 08:24:35.263766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 08:24:35.270053 12351 catalog_manager.cc:1347] Generated new cluster ID: 5df770b5400e42d0956ed508ce7881e4
I20260808 08:24:35.270306 12351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:35.290709 12351 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:35.291754 12351 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:35.303447 12351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3f454d712f8459a92976aaa659fd81e: Generated new TSK 0
I20260808 08:24:35.304482 12351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:35.312096 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12221
W20260808 08:24:35.328137 12202 connection.cc:535] client connection to 127.12.13.62:35641 recv error: Network error: failed to read from TLS socket (remote: 127.12.13.62:35641): Connection reset by peer (error 104)
2026-08-08T08:24:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1718 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:35 dist-test-slave-6ntk krb5kdc[12372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:35 dist-test-slave-6ntk krb5kdc[12372](info): set up 2 sockets
Aug 08 08:24:35 dist-test-slave-6ntk krb5kdc[12372](info): commencing operation
krb5kdc: starting...
W20260808 08:24:36.608116 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:36 dist-test-slave-6ntk krb5kdc[12372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:36.754369 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35207
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:45047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:35207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:36.877144 12390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:36.877498 12390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:36.877576 12390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:36.881860 12390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:36.882014 12390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:36.882055 12390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:36.882092 12390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:36.882130 12390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:36.887610 12390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:45047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:35207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:35207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:36.888767 12390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:36.889846 12390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:36.898871 12400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:36.898918 12390 server_base.cc:1034] running on GCE node
W20260808 08:24:36.898924 12403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:36.899134 12407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:36.899819 12390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:36.901029 12390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:36.902200 12390 hybrid_clock.cc:648] HybridClock initialized: now 1786177476902177 us; error 35 us; skew 500 ppm
Aug 08 08:24:36 dist-test-slave-6ntk krb5kdc[12372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177476, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:36.909262 12390 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:36.910465 12390 webserver.cc:460] Webserver started at http://127.12.13.62:45149/ using document root <none> and password file <none>
I20260808 08:24:36.911095 12390 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:36.911162 12390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:36.911507 12390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:36.913815 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91"
format_stamp: "Formatted at 2026-08-08 08:24:36 on dist-test-slave-6ntk"
server_key: "9f845fc56b5d8f9a7c0e716b1a6e0ccc"
server_key_iv: "25c41ea765493a9ab6426ba24e5e2e62"
server_key_version: "encryptionkey@0"
I20260808 08:24:36.914502 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91"
format_stamp: "Formatted at 2026-08-08 08:24:36 on dist-test-slave-6ntk"
server_key: "9f845fc56b5d8f9a7c0e716b1a6e0ccc"
server_key_iv: "25c41ea765493a9ab6426ba24e5e2e62"
server_key_version: "encryptionkey@0"
I20260808 08:24:36.918481 12390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:36.921108 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:36.922516 12390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:36.922675 12390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91"
format_stamp: "Formatted at 2026-08-08 08:24:36 on dist-test-slave-6ntk"
server_key: "9f845fc56b5d8f9a7c0e716b1a6e0ccc"
server_key_iv: "25c41ea765493a9ab6426ba24e5e2e62"
server_key_version: "encryptionkey@0"
I20260808 08:24:36.922797 12390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:36.927855 12390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:36.931340 12390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:36.931568 12390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:36.939409 12390 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:35207
I20260808 08:24:36.939513 12483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:35207 every 8 connection(s)
I20260808 08:24:36.940734 12390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:36.942477 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12390
I20260808 08:24:36.942652 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:36.942998 12340 external_mini_cluster.cc:1428] Setting key b5ae75ef4177a5b056245b41304426e6
I20260808 08:24:36.944317 12484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:36.949124 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:36.944568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49497 (local address 127.12.13.62:35207)
0808 08:24:36.945024 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:36.945044 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:36.945089 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:24:36.946012 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:36.946038 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:36.946418 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:36.946794 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:36.947616 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:36.947622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:36.947640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:36.947653 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:36.947725 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:36.947753 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:36.947915 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:36.947988 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 08:24:36.948150 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260808 08:24:36.951829 12484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91: Bootstrap starting.
I20260808 08:24:36.952243 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:36.954350 12484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:36.955293 12484 log.cc:824] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:36.956275 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:36.954458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57181 (local address 127.12.13.62:35207)
0808 08:24:36.954583 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:36.954589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:36.954800 (+ 211us) server_negotiation.cc:407] Connection header received
0808 08:24:36.954894 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:36.954897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:36.954937 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:36.955026 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:36.955818 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:36.955821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:36.955823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:36.955841 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:36.955865 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:36.955875 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:36.955946 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:36.956087 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 08:24:36.956121 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 08:24:36.957698 12484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91: No bootstrap required, opened a new log
I20260808 08:24:36.961535 12484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35207 } }
I20260808 08:24:36.961783 12484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:36.961819 12484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d48b7ac28b34250a5d77ec2e6cdcc91, State: Initialized, Role: FOLLOWER
I20260808 08:24:36.962309 12484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35207 } }
I20260808 08:24:36.962450 12484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:36.962498 12484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:36.962574 12484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:36.963491 12484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35207 } }
I20260808 08:24:36.963809 12484 leader_election.cc:302] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d48b7ac28b34250a5d77ec2e6cdcc91; no voters:
I20260808 08:24:36.964085 12484 leader_election.cc:288] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:36.964229 12495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:36.964488 12495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [term 1 LEADER]: Becoming Leader. State: Replica: 8d48b7ac28b34250a5d77ec2e6cdcc91, State: Running, Role: LEADER
I20260808 08:24:36.964840 12495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35207 } }
I20260808 08:24:36.965451 12484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:36.966979 12496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35207 } } }
I20260808 08:24:36.967145 12496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:36.966981 12497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d48b7ac28b34250a5d77ec2e6cdcc91. Latest consensus state: current_term: 1 leader_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b7ac28b34250a5d77ec2e6cdcc91" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 35207 } } }
I20260808 08:24:36.967413 12497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91 [sys.catalog]: This master's current role is: LEADER
W20260808 08:24:36.971138 12514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:36.971246 12514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:36.971323 12509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:36.972064 12509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:36.978593 12509 catalog_manager.cc:1347] Generated new cluster ID: 3c4f201fbbed4bd2aed08104e1af552f
I20260808 08:24:36.978679 12509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:36.993441 12509 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:36.993824 12509 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:37.003001 12509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d48b7ac28b34250a5d77ec2e6cdcc91: Generated new TSK 0
I20260808 08:24:37.003960 12509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:37.008209 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12390
2026-08-08T08:24:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1692 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:37 dist-test-slave-6ntk krb5kdc[12520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:37 dist-test-slave-6ntk krb5kdc[12520](info): set up 2 sockets
Aug 08 08:24:37 dist-test-slave-6ntk krb5kdc[12520](info): commencing operation
krb5kdc: starting...
W20260808 08:24:38.272699 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:38 dist-test-slave-6ntk krb5kdc[12520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:38.410270 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:38465
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:33731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:38465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:38.531726 12541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:38.532030 12541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:38.532109 12541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:38.536330 12541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:38.536406 12541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:38.536427 12541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:38.536443 12541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:38.536509 12541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:38.542098 12541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:33731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:38465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:38465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:38.543344 12541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:38.544481 12541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:38.554414 12548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:38.554441 12550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:38.554421 12547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:38.555586 12541 server_base.cc:1034] running on GCE node
I20260808 08:24:38.556236 12541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:38.557545 12541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:38.558748 12541 hybrid_clock.cc:648] HybridClock initialized: now 1786177478558699 us; error 60 us; skew 500 ppm
Aug 08 08:24:38 dist-test-slave-6ntk krb5kdc[12520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:38.561931 12541 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:38.563042 12541 webserver.cc:460] Webserver started at http://127.12.13.62:46785/ using document root <none> and password file <none>
I20260808 08:24:38.563719 12541 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:38.563822 12541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:38.564073 12541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:38.566449 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "864220f07ad04ee9a1d5c85e258684dd"
format_stamp: "Formatted at 2026-08-08 08:24:38 on dist-test-slave-6ntk"
server_key: "4af44850eb192e483e33e1d0673710cd"
server_key_iv: "7477ff33aa07ad8ddb53fcc62bf1cea0"
server_key_version: "encryptionkey@0"
I20260808 08:24:38.567076 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "864220f07ad04ee9a1d5c85e258684dd"
format_stamp: "Formatted at 2026-08-08 08:24:38 on dist-test-slave-6ntk"
server_key: "4af44850eb192e483e33e1d0673710cd"
server_key_iv: "7477ff33aa07ad8ddb53fcc62bf1cea0"
server_key_version: "encryptionkey@0"
I20260808 08:24:38.571229 12541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:38.573953 12560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:38.575312 12541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:38.575519 12541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "864220f07ad04ee9a1d5c85e258684dd"
format_stamp: "Formatted at 2026-08-08 08:24:38 on dist-test-slave-6ntk"
server_key: "4af44850eb192e483e33e1d0673710cd"
server_key_iv: "7477ff33aa07ad8ddb53fcc62bf1cea0"
server_key_version: "encryptionkey@0"
I20260808 08:24:38.575651 12541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:38.585456 12541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:38.588629 12541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:38.588864 12541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:38.596571 12541 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:38465
I20260808 08:24:38.596587 12631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:38465 every 8 connection(s)
I20260808 08:24:38.597671 12541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:38.598075 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12541
I20260808 08:24:38.598228 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:38.598601 12340 external_mini_cluster.cc:1428] Setting key 60de627ac13304621419cbfa4d1d3ae7
I20260808 08:24:38.601191 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:38.607339 12633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd: Bootstrap starting.
Aug 08 08:24:38 dist-test-slave-6ntk krb5kdc[12520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:38.609833 12633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:38.610958 12633 log.cc:824] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:38.613276 12633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd: No bootstrap required, opened a new log
I20260808 08:24:38.614302 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:38.600103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46107 (local address 127.12.13.62:38465)
0808 08:24:38.600563 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:38.600579 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:38.600615 (+ 36us) server_negotiation.cc:407] Connection header received
0808 08:24:38.601260 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:38.601275 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:38.601698 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:38.602101 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:38.602978 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:38.604010 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:38.604916 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:38.605413 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:38.605600 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:38.608528 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:38.608533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:38.608548 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:38.608578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:38.610781 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:38.611390 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:38.611395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:38.611400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:38.611520 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:38.611870 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:38.611873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:38.611874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:38.612309 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:38.612537 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:38.612828 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 08:24:38.613164 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":149,"threads_started":1}
I20260808 08:24:38.616739 12633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864220f07ad04ee9a1d5c85e258684dd" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38465 } }
I20260808 08:24:38.617004 12633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:38.617103 12633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864220f07ad04ee9a1d5c85e258684dd, State: Initialized, Role: FOLLOWER
I20260808 08:24:38.617388 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:38.617630 12633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864220f07ad04ee9a1d5c85e258684dd" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38465 } }
I20260808 08:24:38.617794 12633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:38.617911 12633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:38.618045 12633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:38.619174 12633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864220f07ad04ee9a1d5c85e258684dd" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38465 } }
I20260808 08:24:38.619613 12633 leader_election.cc:302] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 864220f07ad04ee9a1d5c85e258684dd; no voters:
I20260808 08:24:38.620078 12633 leader_election.cc:288] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:38.620177 12646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:38.620355 12646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [term 1 LEADER]: Becoming Leader. State: Replica: 864220f07ad04ee9a1d5c85e258684dd, State: Running, Role: LEADER
I20260808 08:24:38.620759 12646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864220f07ad04ee9a1d5c85e258684dd" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38465 } }
I20260808 08:24:38.621573 12633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:38.622574 12649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 864220f07ad04ee9a1d5c85e258684dd. Latest consensus state: current_term: 1 leader_uuid: "864220f07ad04ee9a1d5c85e258684dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864220f07ad04ee9a1d5c85e258684dd" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38465 } } }
I20260808 08:24:38.623080 12649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:38.622932 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "864220f07ad04ee9a1d5c85e258684dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864220f07ad04ee9a1d5c85e258684dd" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 38465 } } }
I20260808 08:24:38.623453 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:38.624161 12658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:38.628255 12658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:38.630201 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:38.619924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58793 (local address 127.12.13.62:38465)
0808 08:24:38.620064 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:38.620068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:38.620270 (+ 202us) server_negotiation.cc:407] Connection header received
0808 08:24:38.620527 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:38.620530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:38.620578 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:38.620627 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:38.621613 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:38.622413 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:38.623645 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:38.623999 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:38.624107 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:38.626166 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:38.626170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:38.626172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:38.626202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:38.627992 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:38.628583 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:38.628587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:38.628589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:38.628707 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:38.629067 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:38.629070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:38.629072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:38.629333 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:38.629473 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:38.629799 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 08:24:38.630031 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 08:24:38.634837 12658 catalog_manager.cc:1347] Generated new cluster ID: 996d2858c32146dd9363c1945537043b
I20260808 08:24:38.634924 12658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:38.675202 12658 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:38.676242 12658 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:38.682487 12658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 864220f07ad04ee9a1d5c85e258684dd: Generated new TSK 0
I20260808 08:24:38.683320 12658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:38.694945 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12541
2026-08-08T08:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1686 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:38 dist-test-slave-6ntk krb5kdc[12678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:38 dist-test-slave-6ntk krb5kdc[12678](info): set up 2 sockets
Aug 08 08:24:38 dist-test-slave-6ntk krb5kdc[12678](info): commencing operation
krb5kdc: starting...
W20260808 08:24:39.954748 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:40 dist-test-slave-6ntk krb5kdc[12678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:40.097555 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36133
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:40.219571 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:40.219880 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:40.219924 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:40.224246 12696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:40.224318 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:40.224339 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:40.224356 12696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:40.224371 12696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:40.229794 12696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:36133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:40.231082 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:40.232194 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 08:24:40.241668 12696 server_base.cc:1034] running on GCE node
W20260808 08:24:40.241642 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:40.241678 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:40.242027 12705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:40.242638 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:40.243804 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:40.245009 12696 hybrid_clock.cc:648] HybridClock initialized: now 1786177480244960 us; error 69 us; skew 500 ppm
Aug 08 08:24:40 dist-test-slave-6ntk krb5kdc[12678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:40.247984 12696 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:40.248911 12696 webserver.cc:460] Webserver started at http://127.12.13.62:39823/ using document root <none> and password file <none>
I20260808 08:24:40.249521 12696 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:40.249616 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:40.249890 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:40.252074 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "d935bcc033e14781b7cd6f7c958b9abf"
format_stamp: "Formatted at 2026-08-08 08:24:40 on dist-test-slave-6ntk"
server_key: "66e3245560040661232e06925593d961"
server_key_iv: "dbfc933c640a545fe7b7d7559d552067"
server_key_version: "encryptionkey@0"
I20260808 08:24:40.252673 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "d935bcc033e14781b7cd6f7c958b9abf"
format_stamp: "Formatted at 2026-08-08 08:24:40 on dist-test-slave-6ntk"
server_key: "66e3245560040661232e06925593d961"
server_key_iv: "dbfc933c640a545fe7b7d7559d552067"
server_key_version: "encryptionkey@0"
I20260808 08:24:40.256675 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 08:24:40.259402 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:40.260681 12696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 08:24:40.260864 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "d935bcc033e14781b7cd6f7c958b9abf"
format_stamp: "Formatted at 2026-08-08 08:24:40 on dist-test-slave-6ntk"
server_key: "66e3245560040661232e06925593d961"
server_key_iv: "dbfc933c640a545fe7b7d7559d552067"
server_key_version: "encryptionkey@0"
I20260808 08:24:40.260998 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:40.277264 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:40.281412 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:40.281641 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:40.289497 12696 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:36133
I20260808 08:24:40.289520 12799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:36133 every 8 connection(s)
I20260808 08:24:40.290934 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:40.295085 12800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:40.295598 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12696
I20260808 08:24:40.295748 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:40.296116 12340 external_mini_cluster.cc:1428] Setting key 4cc90e7f4a2e2c4b09042cb87fb9f34b
I20260808 08:24:40.305292 12800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf: Bootstrap starting.
I20260808 08:24:40.307828 12800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:40.308678 12800 log.cc:824] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:40.309551 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:40.297910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52681 (local address 127.12.13.62:36133)
0808 08:24:40.300280 (+ 2370us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:40.300296 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:40.300334 (+ 38us) server_negotiation.cc:407] Connection header received
0808 08:24:40.301297 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:40.301320 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:40.301822 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:40.302257 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:40.303235 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:40.304765 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:40.305925 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:40.306527 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:40.306723 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:40.307641 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:40.307647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:40.307670 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:40.308141 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:40.308202 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:40.308226 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:40.308340 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:40.308476 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 08:24:40.308713 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2204,"thread_start_us":439,"threads_started":1}
I20260808 08:24:40.311122 12800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf: No bootstrap required, opened a new log
I20260808 08:24:40.312716 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:40.315735 12800 raft_consensus.cc:348] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d935bcc033e14781b7cd6f7c958b9abf" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36133 } }
I20260808 08:24:40.316030 12800 raft_consensus.cc:374] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:40.316118 12800 raft_consensus.cc:729] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d935bcc033e14781b7cd6f7c958b9abf, State: Initialized, Role: FOLLOWER
I20260808 08:24:40.316577 12800 consensus_queue.cc:260] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d935bcc033e14781b7cd6f7c958b9abf" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36133 } }
I20260808 08:24:40.316740 12800 raft_consensus.cc:388] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:40.316830 12800 raft_consensus.cc:482] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:40.316955 12800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:40.318002 12800 raft_consensus.cc:504] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d935bcc033e14781b7cd6f7c958b9abf" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36133 } }
I20260808 08:24:40.318423 12800 leader_election.cc:302] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d935bcc033e14781b7cd6f7c958b9abf; no voters:
I20260808 08:24:40.318850 12800 leader_election.cc:288] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:40.318969 12814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:40.319196 12814 raft_consensus.cc:686] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [term 1 LEADER]: Becoming Leader. State: Replica: d935bcc033e14781b7cd6f7c958b9abf, State: Running, Role: LEADER
I20260808 08:24:40.319583 12814 consensus_queue.cc:237] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d935bcc033e14781b7cd6f7c958b9abf" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36133 } }
I20260808 08:24:40.320188 12800 sys_catalog.cc:564] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:40.320914 12814 sys_catalog.cc:455] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d935bcc033e14781b7cd6f7c958b9abf. Latest consensus state: current_term: 1 leader_uuid: "d935bcc033e14781b7cd6f7c958b9abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d935bcc033e14781b7cd6f7c958b9abf" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36133 } } }
I20260808 08:24:40.321099 12814 sys_catalog.cc:458] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:40.321406 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d935bcc033e14781b7cd6f7c958b9abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d935bcc033e14781b7cd6f7c958b9abf" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 36133 } } }
I20260808 08:24:40.321566 12826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:40.321580 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:40.322988 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:40.314918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56303 (local address 127.12.13.62:36133)
0808 08:24:40.315038 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:40.315042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:40.318198 (+ 3156us) server_negotiation.cc:407] Connection header received
0808 08:24:40.318347 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:40.318350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:40.318398 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:40.318469 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:40.318840 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:40.319648 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:40.320858 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:40.321230 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:40.321798 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:40.322343 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:40.322346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:40.322349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:40.322362 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:40.322391 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:40.322404 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:40.322468 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:40.322794 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 08:24:40.322857 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 08:24:40.325960 12826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:40.332867 12826 catalog_manager.cc:1347] Generated new cluster ID: 3c02b33fd3bb4e11a6e94f01096fc774
I20260808 08:24:40.333002 12826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:40.344882 12826 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:40.346110 12826 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:40.352617 12826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d935bcc033e14781b7cd6f7c958b9abf: Generated new TSK 0
I20260808 08:24:40.353402 12826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:40.361399 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12696
2026-08-08T08:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1676 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:40 dist-test-slave-6ntk krb5kdc[12843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:40 dist-test-slave-6ntk krb5kdc[12843](info): set up 2 sockets
Aug 08 08:24:40 dist-test-slave-6ntk krb5kdc[12843](info): commencing operation
krb5kdc: starting...
W20260808 08:24:41.650745 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.007s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:41 dist-test-slave-6ntk krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:41.793176 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42129
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:42129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:41.918180 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:41.918566 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:41.918658 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:41.922931 12859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:41.923030 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:41.923069 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:41.923122 12859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:41.923166 12859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:41.928740 12859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:42129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:42129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:41.930096 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:41.931255 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:41.941020 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:41.941020 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:41.941020 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:41.941689 12859 server_base.cc:1034] running on GCE node
I20260808 08:24:41.942445 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:41.943765 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:41.945016 12859 hybrid_clock.cc:648] HybridClock initialized: now 1786177481944929 us; error 113 us; skew 500 ppm
Aug 08 08:24:41 dist-test-slave-6ntk krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:41.952404 12859 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:41.953472 12859 webserver.cc:460] Webserver started at http://127.12.13.62:33383/ using document root <none> and password file <none>
I20260808 08:24:41.954134 12859 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:41.954200 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:41.954396 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:41.956562 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "6c257a4016ed44f589d5e634e50b8b65"
format_stamp: "Formatted at 2026-08-08 08:24:41 on dist-test-slave-6ntk"
server_key: "cd34925e48b3d360d78aa8b09cbfdcad"
server_key_iv: "c3b6df10fd6da2c25abb55745c1fe9e8"
server_key_version: "encryptionkey@0"
I20260808 08:24:41.957095 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "6c257a4016ed44f589d5e634e50b8b65"
format_stamp: "Formatted at 2026-08-08 08:24:41 on dist-test-slave-6ntk"
server_key: "cd34925e48b3d360d78aa8b09cbfdcad"
server_key_iv: "c3b6df10fd6da2c25abb55745c1fe9e8"
server_key_version: "encryptionkey@0"
I20260808 08:24:41.961100 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:41.963809 12886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:41.965124 12859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 08:24:41.965237 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "6c257a4016ed44f589d5e634e50b8b65"
format_stamp: "Formatted at 2026-08-08 08:24:41 on dist-test-slave-6ntk"
server_key: "cd34925e48b3d360d78aa8b09cbfdcad"
server_key_iv: "c3b6df10fd6da2c25abb55745c1fe9e8"
server_key_version: "encryptionkey@0"
I20260808 08:24:41.965332 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:41.970397 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:41.973567 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:41.973757 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:41.981506 12859 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:42129
I20260808 08:24:41.981539 12958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:42129 every 8 connection(s)
I20260808 08:24:41.982699 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:41.985680 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:41.991544 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 12859
I20260808 08:24:41.991706 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:41.991849 12959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65: Bootstrap starting.
I20260808 08:24:41.992024 12340 external_mini_cluster.cc:1428] Setting key e71eb8746299f94afda0829ab695f687
I20260808 08:24:41.994642 12959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:41.995489 12959 log.cc:824] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:41.997597 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:41.993434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46237 (local address 127.12.13.62:42129)
0808 08:24:41.993920 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:41.993943 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:41.993991 (+ 48us) server_negotiation.cc:407] Connection header received
0808 08:24:41.994670 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:41.994713 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:41.995041 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:41.995329 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:41.996255 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:41.996262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:41.996281 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:41.996298 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:41.996369 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:41.996407 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:41.996500 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:41.996664 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 08:24:41.996854 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260808 08:24:41.997746 12959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65: No bootstrap required, opened a new log
I20260808 08:24:42.000847 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:42.002086 12959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c257a4016ed44f589d5e634e50b8b65" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42129 } }
I20260808 08:24:42.002386 12959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 08:24:42.002451 12959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c257a4016ed44f589d5e634e50b8b65, State: Initialized, Role: FOLLOWER
I20260808 08:24:42.002885 12959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c257a4016ed44f589d5e634e50b8b65" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42129 } }
I20260808 08:24:42.003047 12959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 08:24:42.003119 12959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 08:24:42.003260 12959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:42.004225 12959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c257a4016ed44f589d5e634e50b8b65" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42129 } }
I20260808 08:24:42.004578 12959 leader_election.cc:302] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c257a4016ed44f589d5e634e50b8b65; no voters:
I20260808 08:24:42.004889 12959 leader_election.cc:288] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 08:24:42.005030 12970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:42.005273 12970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [term 1 LEADER]: Becoming Leader. State: Replica: 6c257a4016ed44f589d5e634e50b8b65, State: Running, Role: LEADER
I20260808 08:24:42.005614 12970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c257a4016ed44f589d5e634e50b8b65" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42129 } }
I20260808 08:24:42.006232 12959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:42.007484 12974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c257a4016ed44f589d5e634e50b8b65. Latest consensus state: current_term: 1 leader_uuid: "6c257a4016ed44f589d5e634e50b8b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c257a4016ed44f589d5e634e50b8b65" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42129 } } }
I20260808 08:24:42.007640 12974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:42.007797 12972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c257a4016ed44f589d5e634e50b8b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c257a4016ed44f589d5e634e50b8b65" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 42129 } } }
I20260808 08:24:42.007961 12972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65 [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:42.008073 12984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:42.008983 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:42.003195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60013 (local address 127.12.13.62:42129)
0808 08:24:42.003340 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:42.003346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:42.007102 (+ 3756us) server_negotiation.cc:407] Connection header received
0808 08:24:42.007328 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:42.007331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:42.007373 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:42.007426 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:42.008478 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:42.008483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:42.008490 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 08:24:42.008505 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:42.008539 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 08:24:42.008554 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:42.008648 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:42.008800 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 08:24:42.008843 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 08:24:42.011658 12984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:42.018119 12984 catalog_manager.cc:1347] Generated new cluster ID: 2541d292f6fb4b79a903baf368587325
I20260808 08:24:42.018203 12984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:42.038547 12984 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:42.038972 12984 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:42.049157 12984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c257a4016ed44f589d5e634e50b8b65: Generated new TSK 0
I20260808 08:24:42.050184 12984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 08:24:42.061357 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 12859
2026-08-08T08:24:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1692 ms)
[----------] 18 tests from AuthTokenIssuingTest (30576 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:42 dist-test-slave-6ntk krb5kdc[12997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:42 dist-test-slave-6ntk krb5kdc[12997](info): set up 2 sockets
Aug 08 08:24:42 dist-test-slave-6ntk krb5kdc[12997](info): commencing operation
krb5kdc: starting...
W20260808 08:24:43.340263 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:43.489743 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33959
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33959,127.12.13.61:32925,127.12.13.60:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:43.611181 13019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 08:24:43.611581 13019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:43.611660 13019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:43.611742 13019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:43.615823 13019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:43.615926 13019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:43.615964 13019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:43.616000 13019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:43.616034 13019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:43.621639 13019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33959,127.12.13.61:32925,127.12.13.60:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:33959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:43.622926 13019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:43.624084 13019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:43.633944 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:43.634004 13025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:43.634073 13019 server_base.cc:1034] running on GCE node
W20260808 08:24:43.634021 13026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:43.634976 13019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:43.636301 13019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:43.637532 13019 hybrid_clock.cc:648] HybridClock initialized: now 1786177483637504 us; error 43 us; skew 500 ppm
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:43.640681 13019 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:43.641655 13019 webserver.cc:460] Webserver started at http://127.12.13.62:38923/ using document root <none> and password file <none>
I20260808 08:24:43.642328 13019 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:43.642405 13019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:43.642596 13019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:43.644809 13019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f"
format_stamp: "Formatted at 2026-08-08 08:24:43 on dist-test-slave-6ntk"
server_key: "97e25a44ed6535311c355c170d92620a"
server_key_iv: "39f8e95579131a310eb8f6962e1dce4a"
server_key_version: "encryptionkey@0"
I20260808 08:24:43.645344 13019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f"
format_stamp: "Formatted at 2026-08-08 08:24:43 on dist-test-slave-6ntk"
server_key: "97e25a44ed6535311c355c170d92620a"
server_key_iv: "39f8e95579131a310eb8f6962e1dce4a"
server_key_version: "encryptionkey@0"
I20260808 08:24:43.649284 13019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 08:24:43.651850 13042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:43.653119 13019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 08:24:43.653229 13019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f"
format_stamp: "Formatted at 2026-08-08 08:24:43 on dist-test-slave-6ntk"
server_key: "97e25a44ed6535311c355c170d92620a"
server_key_iv: "39f8e95579131a310eb8f6962e1dce4a"
server_key_version: "encryptionkey@0"
I20260808 08:24:43.653318 13019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:43.672590 13019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:43.676187 13019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:43.676366 13019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:43.683904 13019 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:33959
I20260808 08:24:43.684199 13129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:33959 every 8 connection(s)
I20260808 08:24:43.684968 13019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:43.688067 13130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:43.688295 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13019
I20260808 08:24:43.688423 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:43.688735 12340 external_mini_cluster.cc:1428] Setting key bdc8706ec74f1f1b361f763d27b84820
I20260808 08:24:43.692763 13130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.61@KRBTEST.COM" created.
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:43.708707 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.695360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33959 (local address 127.0.0.1:53218)
0808 08:24:43.695961 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.695999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:43.696862 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:43.697589 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:43.697594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:43.698070 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:43.698357 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.698377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.699536 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.699539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.700267 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.700276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.700943 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.700947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.701056 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.701795 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:43.701820 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:43.703693 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:43.705593 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.705603 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.705614 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.705846 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.706214 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.706218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.706220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.706342 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.706957 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:43.706964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:43.707111 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 08:24:43.707280 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 08:24:43.707680 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":118,"thread_start_us":170,"threads_started":1}
I20260808 08:24:43.708707 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.695360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.12.13.62:33959)
0808 08:24:43.695812 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:43.695833 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:43.696109 (+ 276us) server_negotiation.cc:407] Connection header received
0808 08:24:43.697174 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:43.697180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:43.697399 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:43.697589 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:43.698521 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.699404 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.700391 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.700776 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.700957 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.703866 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:43.703871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:43.703882 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:43.703903 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:43.705481 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.705996 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.706002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.706003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.706114 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.706442 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.706446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.706447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.706839 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:43.706990 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:43.707361 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 08:24:43.707625 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260808 08:24:43.711826 13130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:43.712564 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.712288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56374)
0808 08:24:43.712418 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.712497 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 08:24:43.712760 13052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
I20260808 08:24:43.713701 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.713484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56384)
0808 08:24:43.713596 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.713646 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:24:43.713905 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:32925: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab.
Entry for principal kudu/127.12.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab.
I20260808 08:24:43.722149 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:32925
--webserver_interface=127.12.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab
--principal=kudu/127.12.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33959,127.12.13.61:32925,127.12.13.60:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:24:43.763156 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } attempt: 1
I20260808 08:24:43.764215 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.763898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56400)
0808 08:24:43.764063 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.764134 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 08:24:43.765007 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.764788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56406)
0808 08:24:43.764903 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.764951 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:24:43.765162 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:32925: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
I20260808 08:24:43.833349 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } attempt: 2
I20260808 08:24:43.834277 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.833986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56422)
0808 08:24:43.834133 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.834202 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 08:24:43.835091 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.834885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56424)
0808 08:24:43.834992 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.835037 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:24:43.835258 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:32925: Network error: Client connection negotiation failed: client connection to 127.12.13.61:32925: connect: Connection refused (error 111)
W20260808 08:24:43.845520 13143 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 08:24:43.845947 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:43.846024 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:43.846129 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:43.850664 13143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:43.850739 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:43.850792 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:43.850835 13143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:43.850879 13143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:43.856724 13143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33959,127.12.13.61:32925,127.12.13.60:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab
--principal=kudu/127.12.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:32925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:43.858120 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:43.859279 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:43.868841 13150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:43.868932 13149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:43.868875 13153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:43.869063 13143 server_base.cc:1034] running on GCE node
I20260808 08:24:43.869962 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:43.871132 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:43.872399 13143 hybrid_clock.cc:648] HybridClock initialized: now 1786177483872393 us; error 43 us; skew 500 ppm
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:43.875558 13143 init.cc:377] Logged in from keytab as kudu/127.12.13.61@KRBTEST.COM (short username kudu)
I20260808 08:24:43.876602 13143 webserver.cc:460] Webserver started at http://127.12.13.61:36655/ using document root <none> and password file <none>
I20260808 08:24:43.877206 13143 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:43.877269 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:43.877529 13143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:43.879751 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data/instance:
uuid: "a335c1c94b9d4746aab9a50df9414169"
format_stamp: "Formatted at 2026-08-08 08:24:43 on dist-test-slave-6ntk"
server_key: "719e0232361187bce49ccfbf1d3f4f85"
server_key_iv: "b906afc6e5b2a3008ff8c5e30bb39779"
server_key_version: "encryptionkey@0"
I20260808 08:24:43.880424 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal/instance:
uuid: "a335c1c94b9d4746aab9a50df9414169"
format_stamp: "Formatted at 2026-08-08 08:24:43 on dist-test-slave-6ntk"
server_key: "719e0232361187bce49ccfbf1d3f4f85"
server_key_iv: "b906afc6e5b2a3008ff8c5e30bb39779"
server_key_version: "encryptionkey@0"
I20260808 08:24:43.884392 13143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:43.886961 13164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:43.888218 13143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:24:43.888372 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal
uuid: "a335c1c94b9d4746aab9a50df9414169"
format_stamp: "Formatted at 2026-08-08 08:24:43 on dist-test-slave-6ntk"
server_key: "719e0232361187bce49ccfbf1d3f4f85"
server_key_iv: "b906afc6e5b2a3008ff8c5e30bb39779"
server_key_version: "encryptionkey@0"
I20260808 08:24:43.888484 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:43.901612 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:43.905087 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:43.905292 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:43.913178 13143 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.61:32925
I20260808 08:24:43.913494 13241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.61:32925 every 8 connection(s)
I20260808 08:24:43.914345 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
I20260808 08:24:43.917258 13242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:43.920748 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13143
I20260808 08:24:43.920855 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-1/wal/instance
I20260808 08:24:43.921150 12340 external_mini_cluster.cc:1428] Setting key 5bb428181c3bad96ceb6e595371565af
I20260808 08:24:43.921271 13242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:43.926442 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } attempt: 3
WARNING: no policy specified for kudu/127.12.13.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.60@KRBTEST.COM" created.
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:43.940150 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.926889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56432)
0808 08:24:43.927069 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.927088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:43.927154 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:43.927828 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:43.927831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:43.928054 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:43.928169 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.928175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.929706 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.929716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.930434 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.930445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.931288 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.931291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.931412 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.931992 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:43.932004 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:43.935612 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:43.937951 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.937955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.937957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.938165 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.938649 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.938655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.938660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.938789 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.939579 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:43.939582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:43.939695 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 08:24:43.939807 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 08:24:43.939968 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"spinlock_wait_cycles":768}
I20260808 08:24:43.940629 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.923300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.12.13.62:33959)
0808 08:24:43.923516 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:43.923523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:43.924419 (+ 896us) server_negotiation.cc:407] Connection header received
0808 08:24:43.925373 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:43.925387 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:43.925462 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:43.925572 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:43.927037 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.927853 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.929543 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.929992 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.930133 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.935186 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:43.935196 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:43.935215 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:43.935251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:43.937763 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.938728 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.938732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.938733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.938818 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.939198 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.939204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.939208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.939488 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:43.939597 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:43.940215 (+ 618us) server_negotiation.cc:299] Negotiation successful
0808 08:24:43.940377 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 08:24:43.941574 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.923700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33959 (local address 127.0.0.1:53222)
0808 08:24:43.924222 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.924276 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:43.925151 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:43.925725 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:43.925731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:43.926160 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:43.926864 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.926887 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.928007 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.928010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.929402 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.929413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.930160 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.930164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.930362 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.931077 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:43.931093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:43.935017 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:43.938294 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.938301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.938308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.938632 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.938940 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.938943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.938945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.939082 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.939669 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:43.939679 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:43.939899 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 08:24:43.940775 (+ 876us) client_negotiation.cc:240] Negotiation successful
0808 08:24:43.941025 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":154,"threads_started":1}
I20260808 08:24:43.941571 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.926995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.12.13.61:32925)
0808 08:24:43.927378 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:43.927390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:43.927412 (+ 22us) server_negotiation.cc:407] Connection header received
0808 08:24:43.927452 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:43.927457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:43.927653 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:43.928313 (+ 660us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:43.928435 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.929598 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.930572 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.931156 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.931245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.935795 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:43.935806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:43.935826 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:43.935857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:43.937772 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.938287 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.938294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.938302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.938512 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.938910 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.938916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.938920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.939466 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:43.939617 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:43.939968 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 08:24:43.940320 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":281,"spinlock_wait_cycles":1280,"thread_start_us":173,"threads_started":1}
I20260808 08:24:43.943281 13242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:43.944003 13130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:43.944774 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.944464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53342)
0808 08:24:43.944587 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.944642 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 08:24:43.945688 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.945326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53358)
0808 08:24:43.945557 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.945623 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260808 08:24:43.945920 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Aug 08 08:24:43 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:43.952813 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.944119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.12.13.61:32925)
0808 08:24:43.944226 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:43.944230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:43.944244 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:24:43.944356 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:43.944362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:43.944430 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:43.944510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:43.945120 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.945814 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.946695 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.946995 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.947102 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.949385 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:43.949388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:43.949390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:43.949413 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:43.951032 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.951541 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.951544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.951545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.951621 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:43.951915 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:43.951917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:43.951918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:43.952105 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:43.952185 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:43.952554 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 08:24:43.952689 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":43}
I20260808 08:24:43.952796 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.943895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56446)
0808 08:24:43.944146 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.944161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:43.944254 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:43.944630 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:43.944634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:43.944831 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:43.944980 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.944992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.945997 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.946002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.946597 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:43.946606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:43.947121 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:43.947124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:43.947202 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:43.947687 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:43.947699 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:43.949168 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:43.951167 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.951170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.951171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.951365 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.951721 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:43.951723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:43.951724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:43.951828 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:43.952210 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:43.952212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:43.952342 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 08:24:43.952497 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 08:24:43.952624 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260808 08:24:43.953759 13242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:43.954718 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.954323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53370)
0808 08:24:43.954478 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.954618 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260808 08:24:43.955420 13171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab.
Entry for principal kudu/127.12.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab.
I20260808 08:24:43.956423 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.956179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53386)
0808 08:24:43.956290 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.956339 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:24:43.956626 13242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
I20260808 08:24:43.957080 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:42383
--webserver_interface=127.12.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab
--principal=kudu/127.12.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:33959,127.12.13.61:32925,127.12.13.60:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:24:43.977094 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 1
I20260808 08:24:43.978067 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.977703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53392)
0808 08:24:43.977864 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.977979 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 08:24:43.978806 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:43.978586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53408)
0808 08:24:43.978699 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:43.978752 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 08:24:43.978957 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
I20260808 08:24:44.005983 13242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 1
I20260808 08:24:44.006922 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.006611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53412)
0808 08:24:44.006772 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.006837 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 08:24:44.007723 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.007502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53418)
0808 08:24:44.007617 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.007671 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:24:44.007846 13242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
I20260808 08:24:44.054147 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 2
I20260808 08:24:44.055106 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.054758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53424)
0808 08:24:44.054941 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.055018 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 08:24:44.055876 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.055647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53430)
0808 08:24:44.055758 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.055806 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 08:24:44.056025 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
I20260808 08:24:44.076031 13242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 2
I20260808 08:24:44.076995 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.076645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53438)
0808 08:24:44.076837 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.076909 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 08:24:44.077765 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.077530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53454)
0808 08:24:44.077651 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.077706 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 08:24:44.077950 13242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
W20260808 08:24:44.079156 13252 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 08:24:44.079499 13252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:44.079560 13252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:44.079639 13252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:44.083830 13252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:44.083916 13252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:44.083953 13252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:44.084007 13252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:44.084045 13252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:44.089572 13252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:33959,127.12.13.61:32925,127.12.13.60:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab
--principal=kudu/127.12.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:44.090809 13252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:44.091943 13252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:44.101080 13260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:44.101348 13252 server_base.cc:1034] running on GCE node
W20260808 08:24:44.101467 13262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:44.101394 13269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:44.102181 13252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:44.103368 13252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:44.104545 13252 hybrid_clock.cc:648] HybridClock initialized: now 1786177484104512 us; error 49 us; skew 500 ppm
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:44.107646 13252 init.cc:377] Logged in from keytab as kudu/127.12.13.60@KRBTEST.COM (short username kudu)
I20260808 08:24:44.108632 13252 webserver.cc:460] Webserver started at http://127.12.13.60:37151/ using document root <none> and password file <none>
I20260808 08:24:44.109201 13252 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:44.109263 13252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:44.109442 13252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:44.111688 13252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data/instance:
uuid: "dc55aad74f1b4ae8b603b2441753e76b"
format_stamp: "Formatted at 2026-08-08 08:24:44 on dist-test-slave-6ntk"
server_key: "ad4852c0185c2814af362b2c90ecfbcc"
server_key_iv: "e9e49f076c64d01fbc2856e46dd500ef"
server_key_version: "encryptionkey@0"
I20260808 08:24:44.112247 13252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal/instance:
uuid: "dc55aad74f1b4ae8b603b2441753e76b"
format_stamp: "Formatted at 2026-08-08 08:24:44 on dist-test-slave-6ntk"
server_key: "ad4852c0185c2814af362b2c90ecfbcc"
server_key_iv: "e9e49f076c64d01fbc2856e46dd500ef"
server_key_version: "encryptionkey@0"
I20260808 08:24:44.116367 13252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:44.119177 13279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:44.120592 13252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 08:24:44.120757 13252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal
uuid: "dc55aad74f1b4ae8b603b2441753e76b"
format_stamp: "Formatted at 2026-08-08 08:24:44 on dist-test-slave-6ntk"
server_key: "ad4852c0185c2814af362b2c90ecfbcc"
server_key_iv: "e9e49f076c64d01fbc2856e46dd500ef"
server_key_version: "encryptionkey@0"
I20260808 08:24:44.120896 13252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:44.148803 13252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:44.152549 13252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:44.152778 13252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:44.155191 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 3
I20260808 08:24:44.156031 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.155688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53470)
0808 08:24:44.155863 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.155929 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 08:24:44.156919 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.156631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53482)
0808 08:24:44.156789 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.156840 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260808 08:24:44.157224 13130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:42383: Network error: Client connection negotiation failed: client connection to 127.12.13.60:42383: connect: Connection refused (error 111)
I20260808 08:24:44.160948 13252 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.60:42383
I20260808 08:24:44.161242 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.60:42383 every 8 connection(s)
I20260808 08:24:44.162065 13252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
I20260808 08:24:44.164987 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13252
I20260808 08:24:44.165131 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786177179845590-12340-0/minicluster-data/master-2/wal/instance
I20260808 08:24:44.165410 13352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:44.165431 12340 external_mini_cluster.cc:1428] Setting key 876278ea3276023e851c0106bac6d1e6
I20260808 08:24:44.169150 13242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 3
I20260808 08:24:44.171327 13352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
I20260808 08:24:44.181136 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.169646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53510)
0808 08:24:44.169767 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.169783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:44.169893 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:44.170424 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:44.170428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:44.170609 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:44.170702 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.170707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.171747 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.171751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.172300 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.172309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.173159 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.173163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.173270 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.173923 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:44.173938 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:44.175649 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:44.179436 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.179438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.179440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.179680 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.180045 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.180049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.180050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.180145 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.180585 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:44.180588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:44.180702 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 08:24:44.180807 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 08:24:44.180959 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:44.182547 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.169702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.12.13.60:42383)
0808 08:24:44.170116 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.170125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.170140 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:24:44.170227 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.170231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.170282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.170395 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.170819 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.171620 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.172431 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.172977 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.173176 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.175830 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.175838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.175852 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.175900 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.179012 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.179776 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.179779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.179779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.179917 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.180264 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.180267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.180269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.180467 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.181230 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.181319 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.182008 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":576,"server-negotiator.queue_time_us":340,"spinlock_wait_cycles":6016,"thread_start_us":159,"threads_started":1}
I20260808 08:24:44.182567 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.166929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.12.13.60:42383)
0808 08:24:44.167363 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.167383 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.167428 (+ 45us) server_negotiation.cc:407] Connection header received
0808 08:24:44.168404 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.168431 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.168870 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.169273 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.170315 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.171703 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.172870 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.173207 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.173293 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.176397 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.176403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.176405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.176433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.179012 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.179422 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.179426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.179431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.179544 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.179852 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.179858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.179862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.180405 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.180622 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.181004 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.181529 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":239,"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260808 08:24:44.187058 13242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: Bootstrap starting.
I20260808 08:24:44.189709 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.173690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.12.13.62:33959)
0808 08:24:44.173813 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.173818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.175274 (+ 1456us) server_negotiation.cc:407] Connection header received
0808 08:24:44.175432 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.175435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.175490 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.175583 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.176450 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.177159 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.178231 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.178462 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.178572 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.182079 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.182082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.182084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.182108 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.184475 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.185099 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.185103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.185104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.185204 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.188126 (+ 2922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.188130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.188131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.188518 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.188666 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.189323 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.189512 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 08:24:44.190145 13242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:44.190936 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.173437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:33959 (local address 127.0.0.1:53226)
0808 08:24:44.175118 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.175173 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:44.175306 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:44.175660 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:44.175666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:44.176139 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:44.176309 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.176320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.177283 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.177290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.178112 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.178122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.178577 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.178580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.178701 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.179880 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:44.179901 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:44.181933 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:44.184600 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.184612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.184621 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.184968 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.187790 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.187793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.187795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.187955 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.188763 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:44.188768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:44.188946 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 08:24:44.190356 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0808 08:24:44.190685 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"spinlock_wait_cycles":1280,"thread_start_us":138,"threads_started":1}
I20260808 08:24:44.191138 13242 log.cc:824] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:44.194866 13352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:44.195623 13242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: No bootstrap required, opened a new log
I20260808 08:24:44.197803 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.185313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.12.13.62:33959)
0808 08:24:44.187397 (+ 2084us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.187413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.187428 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:24:44.187463 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.187466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.187507 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.187604 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.188511 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.189344 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.190506 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.190853 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.191375 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.194210 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.194214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.194216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.194243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.196091 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.196561 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.196563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.196565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.196643 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.196965 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.196967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.196968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.197144 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.197287 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.197512 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.197616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1994,"thread_start_us":107,"threads_started":1}
I20260808 08:24:44.200222 13242 raft_consensus.cc:348] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.200522 13242 raft_consensus.cc:729] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a335c1c94b9d4746aab9a50df9414169, State: Initialized, Role: FOLLOWER
I20260808 08:24:44.200891 13242 consensus_queue.cc:260] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.201371 13365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.201610 13365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:44.202749 13242 sys_catalog.cc:564] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:44.207990 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.195384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:32925 (local address 127.0.0.1:56462)
0808 08:24:44.198011 (+ 2627us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.198027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:44.198169 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:44.198486 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:44.198489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:44.198671 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:44.198777 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.198782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.199757 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.199760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.200332 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.200342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.200936 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.200940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.201035 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.201579 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:44.201594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:44.203178 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:44.205668 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.205672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.205674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.206066 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.206442 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.206446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.206449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.206562 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.207228 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:44.207235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:44.207372 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 08:24:44.207513 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 08:24:44.207735 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2536}
I20260808 08:24:44.208657 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.195493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.12.13.61:32925)
0808 08:24:44.195612 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.195617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.198090 (+ 2473us) server_negotiation.cc:407] Connection header received
0808 08:24:44.198297 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.198300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.198354 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.198470 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.198896 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.199632 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.200473 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.200792 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.200898 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.203316 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.203320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.203323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.203348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.205485 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.206209 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.206214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.206218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.206328 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.206690 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.206695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.206697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.207032 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.207157 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.208273 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.208426 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 08:24:44.209460 13377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:44.209533 13377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:44.207569 13143 master_runner.cc:186] Error getting master registration for 127.12.13.62:33959: OK, instance_id { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" instance_seqno: 1786177483681797 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 08:24:44.210983 13143 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 08:24:44.211278 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.199094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.12.13.61:32925)
0808 08:24:44.199400 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.199409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.199422 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:24:44.199455 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.199458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.199494 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.199582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.200334 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.201041 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.202250 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.202498 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.202614 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.206582 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.206588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.206595 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.206628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.209317 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.209825 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.209832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.209836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.209981 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.210305 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.210309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.210311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.210516 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.210828 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.210954 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.211084 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":132,"threads_started":1}
I20260808 08:24:44.212637 13352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:44.216080 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
I20260808 08:24:44.217730 13088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53236
I20260808 08:24:44.223181 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.213223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53518)
0808 08:24:44.213317 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.213328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:44.213403 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:44.213818 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:44.213822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:44.214051 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:44.214163 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.214171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.215074 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.215080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.215872 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.215897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.216611 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.216616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.216711 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.217307 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:44.217320 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:44.219104 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:44.221299 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.221302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.221305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.221574 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.221915 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.221919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.221921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.222023 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.222566 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:44.222569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:44.222681 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 08:24:44.222808 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 08:24:44.222988 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 08:24:44.223301 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.213446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.12.13.60:42383)
0808 08:24:44.213591 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.213595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.213608 (+ 13us) server_negotiation.cc:407] Connection header received
0808 08:24:44.213641 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.213644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.213685 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.213772 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.214288 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.214925 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.216083 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.216468 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.216580 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.219278 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.219281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.219300 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.219332 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.221172 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.221681 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.221686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.221689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.221779 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.222125 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.222128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.222130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.222449 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.222548 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.222898 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.223074 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":54}
I20260808 08:24:44.226967 13352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: Bootstrap starting.
I20260808 08:24:44.230063 13352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:44.231042 13352 log.cc:824] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:44.233464 13352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: No bootstrap required, opened a new log
I20260808 08:24:44.237967 13352 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.238257 13352 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc55aad74f1b4ae8b603b2441753e76b, State: Initialized, Role: FOLLOWER
I20260808 08:24:44.238636 13352 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.239228 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.239383 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:44.240206 13352 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: configured and running, proceeding with master startup.
W20260808 08:24:44.246476 13401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:44.246572 13401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:44.244807 13252 master_runner.cc:186] Error getting master registration for 127.12.13.62:33959: OK, instance_id { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" instance_seqno: 1786177483681797 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 08:24:44.248485 13252 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 08:24:44.321406 13130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } attempt: 4
Aug 08 08:24:44 dist-test-slave-6ntk krb5kdc[12997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
I20260808 08:24:44.332270 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.322288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.12.13.60:42383)
0808 08:24:44.322436 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.322443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.322518 (+ 75us) server_negotiation.cc:407] Connection header received
0808 08:24:44.322736 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.322740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.322799 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.322886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.323463 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.324270 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.325214 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.325602 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.325728 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.328494 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.328499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.328502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.328541 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.330243 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.330823 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.330827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.330831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.330928 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.331280 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.331285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.331287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.331502 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.331632 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.331960 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.332087 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:24:44.332270 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.322188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:42383 (local address 127.0.0.1:53522)
0808 08:24:44.322434 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:44.322453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:44.322584 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:44.322915 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:44.322929 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:44.323183 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:44.323297 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.323305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.324404 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.324408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.325034 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:44.325045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.325930 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.325935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:44.326060 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.326694 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:44.326720 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:44.328343 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:44.330386 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.330391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.330394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.330713 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.331050 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:44.331054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:44.331060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:44.331181 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:44.331620 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:44.331623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:44.331752 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 08:24:44.331875 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 08:24:44.332041 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260808 08:24:44.336570 13130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f: Bootstrap starting.
I20260808 08:24:44.339890 13130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:44.341014 13130 log.cc:824] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:44.343784 13130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f: No bootstrap required, opened a new log
I20260808 08:24:44.347545 13130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.347827 13130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b08e0ef90d14d37a25f4d0d4dcd0e7f, State: Initialized, Role: FOLLOWER
I20260808 08:24:44.348228 13130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.348884 13404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.349210 13404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:44.349844 13130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:44.349941 13088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 08:24:44.350045 13088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:44.351637 13088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.354214 13203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a335c1c94b9d4746aab9a50df9414169"
I20260808 08:24:44.354511 13203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:44.356045 13203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b08e0ef90d14d37a25f4d0d4dcd0e7f in term 1.
I20260808 08:24:44.356680 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dc55aad74f1b4ae8b603b2441753e76b"
I20260808 08:24:44.356968 13316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:44.358387 13316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b08e0ef90d14d37a25f4d0d4dcd0e7f in term 1.
I20260808 08:24:44.359572 13052 leader_election.cc:302] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b08e0ef90d14d37a25f4d0d4dcd0e7f, a335c1c94b9d4746aab9a50df9414169; no voters:
I20260808 08:24:44.352793 13088 leader_election.cc:288] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [CANDIDATE]: Term 1 election: Requested vote from peers a335c1c94b9d4746aab9a50df9414169 (127.12.13.61:32925), dc55aad74f1b4ae8b603b2441753e76b (127.12.13.60:42383)
W20260808 08:24:44.356940 13419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:44.361958 13419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:44.362161 13404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:44.362471 13404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [term 1 LEADER]: Becoming Leader. State: Replica: 6b08e0ef90d14d37a25f4d0d4dcd0e7f, State: Running, Role: LEADER
I20260808 08:24:44.362908 13404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } }
I20260808 08:24:44.365057 13404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b08e0ef90d14d37a25f4d0d4dcd0e7f. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.365239 13404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:44.365830 13431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:44.367096 13019 master_runner.cc:263] No leader master found from master 6b08e0ef90d14d37a25f4d0d4dcd0e7f
I20260808 08:24:44.367183 13019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 08:24:44.368654 13431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:44.379971 13316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [term 1 FOLLOWER]: Refusing update from remote peer 6b08e0ef90d14d37a25f4d0d4dcd0e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:44.380980 13429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:44.386096 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b08e0ef90d14d37a25f4d0d4dcd0e7f. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.386240 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:44.386933 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.366411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.12.13.61:32925)
0808 08:24:44.366775 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.366781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.368668 (+ 1887us) server_negotiation.cc:407] Connection header received
0808 08:24:44.368917 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.368921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.368980 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.369044 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.370307 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.371100 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.372381 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.372776 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.372861 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.375417 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.375424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.375430 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.375465 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.377599 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.382786 (+ 5187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.382790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.382792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.382910 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.383853 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.383857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.383859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.384101 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.384276 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.386506 (+ 2230us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.386678 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260808 08:24:44.387603 13203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [term 1 FOLLOWER]: Refusing update from remote peer 6b08e0ef90d14d37a25f4d0d4dcd0e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:44.388551 13429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:44.389168 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.367086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.12.13.60:42383)
0808 08:24:44.367368 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.367374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.371431 (+ 4057us) server_negotiation.cc:407] Connection header received
0808 08:24:44.371629 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.371633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.371704 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.371764 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.372853 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.373571 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.374889 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.375229 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.375322 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.379251 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.379255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.379259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.379292 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.381171 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.383389 (+ 2218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.383393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.383395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.383494 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.384652 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.384656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.384658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.384891 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.385017 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.388840 (+ 3823us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.388993 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260808 08:24:44.396158 13365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b08e0ef90d14d37a25f4d0d4dcd0e7f. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.396309 13365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:44.397133 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:44.368046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.12.13.62:33959)
0808 08:24:44.368156 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.368162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.375180 (+ 7018us) server_negotiation.cc:407] Connection header received
0808 08:24:44.381335 (+ 6155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.381339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.381396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.381454 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.385391 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.386280 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.389778 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:44.390185 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:44.390279 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:44.392620 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:44.392625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:44.392628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:44.392660 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:44.394741 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.395348 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.395352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.395354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.395457 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:44.395984 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:44.395991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:44.395994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:44.396272 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:44.396396 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:44.396827 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 08:24:44.396964 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 08:24:44.401791 13429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.401999 13429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:44.403499 13431 catalog_manager.cc:1347] Generated new cluster ID: e4d5e282015e43f7ae6183c455423a5b
I20260808 08:24:44.403585 13431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260808 08:24:44.404340 13287 tablet.cc:2369] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 08:24:44.407284 13365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.407446 13365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:44.408089 13429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.408241 13429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:44.411460 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b08e0ef90d14d37a25f4d0d4dcd0e7f" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 33959 } } peers { permanent_uuid: "a335c1c94b9d4746aab9a50df9414169" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 32925 } } peers { permanent_uuid: "dc55aad74f1b4ae8b603b2441753e76b" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 42383 } } }
I20260808 08:24:44.411603 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:44.421061 13440 mvcc.cc:205] Tried to move back new op lower bound from 7316182976011685888 to 7316182975954960384. Current Snapshot: MvccSnapshot[applied={T|T < 7316182976011685888}]
I20260808 08:24:44.439139 13431 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:44.440990 13431 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:44.458022 13431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b08e0ef90d14d37a25f4d0d4dcd0e7f: Generated new TSK 0
I20260808 08:24:44.458976 13431 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 08:24:44.488412 13361 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.486109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.12.13.62:33959)
0808 08:24:44.486259 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.486265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.487200 (+ 935us) server_negotiation.cc:407] Connection header received
0808 08:24:44.487393 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.487397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.487453 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.487513 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.488251 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53266: BlockingRecv error: recv got EOF from 127.0.0.1:53266 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 08:24:44.488799 13358 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.486824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53542 (local address 127.12.13.60:42383)
0808 08:24:44.487063 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.487067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.487818 (+ 751us) server_negotiation.cc:407] Connection header received
0808 08:24:44.488001 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.488005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.488056 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.488107 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.488668 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53542: BlockingRecv error: recv got EOF from 127.0.0.1:53542 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
W20260808 08:24:44.489594 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.486060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.12.13.61:32925)
0808 08:24:44.486195 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.486200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.486933 (+ 733us) server_negotiation.cc:407] Connection header received
0808 08:24:44.487120 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.487123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.487179 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.487261 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.489475 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56474: BlockingRecv error: recv got EOF from 127.0.0.1:56474 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 08:24:44.492941 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.491749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.12.13.61:32925)
0808 08:24:44.491913 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.491917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.492114 (+ 197us) server_negotiation.cc:407] Connection header received
0808 08:24:44.492236 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.492240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.492327 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.492418 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.492796 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56488: BlockingRecv error: recv got EOF from 127.0.0.1:56488 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 08:24:44.496951 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.496115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.12.13.61:32925)
0808 08:24:44.496251 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.496257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.496272 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:24:44.496323 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.496326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.496385 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.496480 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.496849 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56492: BlockingRecv error: recv got EOF from 127.0.0.1:56492 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 08:24:44.501756 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.500710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.12.13.61:32925)
0808 08:24:44.500846 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.500849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.500962 (+ 113us) server_negotiation.cc:407] Connection header received
0808 08:24:44.501079 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.501081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.501122 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.501188 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.501649 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56496: BlockingRecv error: recv got EOF from 127.0.0.1:56496 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 08:24:44.509151 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.507971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.12.13.61:32925)
0808 08:24:44.508133 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.508137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.508248 (+ 111us) server_negotiation.cc:407] Connection header received
0808 08:24:44.508393 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.508396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.508438 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.508508 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.509037 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56504: BlockingRecv error: recv got EOF from 127.0.0.1:56504 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260808 08:24:44.521267 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.519840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.12.13.61:32925)
0808 08:24:44.520015 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.520021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.520226 (+ 205us) server_negotiation.cc:407] Connection header received
0808 08:24:44.520383 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.520388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.520453 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.520569 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.521111 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56520: BlockingRecv error: recv got EOF from 127.0.0.1:56520 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 08:24:44.541172 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.539718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.12.13.61:32925)
0808 08:24:44.539941 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.539948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.540075 (+ 127us) server_negotiation.cc:407] Connection header received
0808 08:24:44.540284 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.540288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.540364 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.540500 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.540998 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56536: BlockingRecv error: recv got EOF from 127.0.0.1:56536 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 08:24:44.576669 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.575380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.12.13.61:32925)
0808 08:24:44.575581 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.575587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.575706 (+ 119us) server_negotiation.cc:407] Connection header received
0808 08:24:44.575825 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.575829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.575899 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.576016 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.576528 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56548: BlockingRecv error: recv got EOF from 127.0.0.1:56548 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 08:24:44.644797 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.643446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.12.13.61:32925)
0808 08:24:44.643678 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.643682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.643792 (+ 110us) server_negotiation.cc:407] Connection header received
0808 08:24:44.643911 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.643914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.643969 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.644058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.644661 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56550: BlockingRecv error: recv got EOF from 127.0.0.1:56550 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260808 08:24:44.776513 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:44.775178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.12.13.61:32925)
0808 08:24:44.775367 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:44.775372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:44.775502 (+ 130us) server_negotiation.cc:407] Connection header received
0808 08:24:44.775689 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:44.775694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:44.775752 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:44.775854 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:44.776400 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56558: BlockingRecv error: recv got EOF from 127.0.0.1:56558 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260808 08:24:45.036465 13363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:45.035286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.12.13.61:32925)
0808 08:24:45.035471 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:45.035475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:45.035512 (+ 37us) server_negotiation.cc:407] Connection header received
0808 08:24:45.035658 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:45.035661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:45.035714 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:45.035807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:45.036296 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56574: BlockingRecv error: recv got EOF from 127.0.0.1:56574 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 08:24:45.211233 13377 catalog_manager.cc:1259] Loaded cluster ID: e4d5e282015e43f7ae6183c455423a5b
I20260808 08:24:45.211380 13377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: loading cluster ID for follower catalog manager: success
I20260808 08:24:45.213240 13377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: acquiring CA information for follower catalog manager: success
I20260808 08:24:45.214573 13377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a335c1c94b9d4746aab9a50df9414169: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:24:45.248093 13401 catalog_manager.cc:1259] Loaded cluster ID: e4d5e282015e43f7ae6183c455423a5b
I20260808 08:24:45.248191 13401 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: loading cluster ID for follower catalog manager: success
I20260808 08:24:45.250000 13401 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: acquiring CA information for follower catalog manager: success
I20260808 08:24:45.251305 13401 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc55aad74f1b4ae8b603b2441753e76b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:24:45.555255 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:45.550904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.12.13.61:32925)
0808 08:24:45.551180 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:45.551188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:45.551315 (+ 127us) server_negotiation.cc:407] Connection header received
0808 08:24:45.551506 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:45.551510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:45.551661 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:45.551788 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:45.552110 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.552658 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.553570 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.553811 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.553995 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:45.554349 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:45.554826 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:45.554981 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:45.555024 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 08:24:45.555083 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260808 08:24:45.562619 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:45.558812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.12.13.60:42383)
0808 08:24:45.559087 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:45.559091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:45.559105 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:24:45.559191 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:45.559193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:45.559314 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:45.559423 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:45.559640 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.560146 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.561082 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.561299 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.561378 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:45.561699 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:45.562149 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:45.562242 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:45.562326 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 08:24:45.562377 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260808 08:24:45.570016 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:45.565974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.12.13.61:32925)
0808 08:24:45.566131 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:45.566135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:45.566297 (+ 162us) server_negotiation.cc:407] Connection header received
0808 08:24:45.566497 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:45.566500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:45.566567 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:45.566647 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:45.567006 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.567830 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.568838 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.569111 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.569216 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:45.569447 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:45.569543 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:45.569611 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:45.569768 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 08:24:45.569826 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 08:24:45.570601 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:45.566527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.12.13.60:42383)
0808 08:24:45.566666 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:45.566673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:45.566996 (+ 323us) server_negotiation.cc:407] Connection header received
0808 08:24:45.567182 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:45.567185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:45.567233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:45.567317 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:45.567551 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.568147 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.569260 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.569602 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.569676 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:45.570075 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:45.570156 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:45.570232 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:45.570376 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 08:24:45.570439 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 08:24:45.570912 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:45.565847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.12.13.62:33959)
0808 08:24:45.566160 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:45.566165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:45.566274 (+ 109us) server_negotiation.cc:407] Connection header received
0808 08:24:45.566445 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:45.566449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:45.566620 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:45.566735 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:45.567137 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.567925 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.569124 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:45.569457 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:45.569561 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:45.569914 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:45.570290 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:45.570392 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:45.570702 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 08:24:45.570770 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":76,"threads_started":1}
I20260808 08:24:45.574031 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13019
I20260808 08:24:45.591728 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13143
I20260808 08:24:45.609613 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13252
2026-08-08T08:24:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3549 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 08:24:45 dist-test-slave-6ntk krb5kdc[13559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 08:24:45 dist-test-slave-6ntk krb5kdc[13559](info): set up 2 sockets
Aug 08 08:24:45 dist-test-slave-6ntk krb5kdc[13559](info): commencing operation
krb5kdc: starting...
W20260808 08:24:46.959441 12340 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/test-user.keytab.
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T08:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T08:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
Entry for principal kudu/127.12.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab.
I20260808 08:24:47.100898 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:34617
--webserver_interface=127.12.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:34617,127.12.13.61:37585,127.12.13.60:32867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 08:24:47.223980 13576 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 08:24:47.224400 13576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:47.224489 13576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:47.224586 13576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:47.229254 13576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:47.229363 13576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:47.229406 13576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:47.229497 13576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:47.229547 13576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:47.235424 13576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:34617,127.12.13.61:37585,127.12.13.60:32867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.62.keytab
--principal=kudu/127.12.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.62:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:47.236651 13576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:47.237821 13576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:47.247071 13587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:47.247078 13583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:47.247078 13584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:47.247184 13576 server_base.cc:1034] running on GCE node
I20260808 08:24:47.248044 13576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:47.249284 13576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:47.250514 13576 hybrid_clock.cc:648] HybridClock initialized: now 1786177487250500 us; error 34 us; skew 500 ppm
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:47.258903 13576 init.cc:377] Logged in from keytab as kudu/127.12.13.62@KRBTEST.COM (short username kudu)
I20260808 08:24:47.259944 13576 webserver.cc:460] Webserver started at http://127.12.13.62:44463/ using document root <none> and password file <none>
I20260808 08:24:47.260600 13576 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:47.260667 13576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:47.260926 13576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:47.263190 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data/instance:
uuid: "6c3bb418cae14ae7ad41f9b0837c3fad"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "c82a7b4a3051893c3932e57a5ae9ed23"
server_key_iv: "5758c1ed5d76063ca9f2c07dcd5b4e46"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.263783 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal/instance:
uuid: "6c3bb418cae14ae7ad41f9b0837c3fad"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "c82a7b4a3051893c3932e57a5ae9ed23"
server_key_iv: "5758c1ed5d76063ca9f2c07dcd5b4e46"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.267778 13576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 08:24:47.270404 13596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:47.271701 13576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:24:47.271840 13576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal
uuid: "6c3bb418cae14ae7ad41f9b0837c3fad"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "c82a7b4a3051893c3932e57a5ae9ed23"
server_key_iv: "5758c1ed5d76063ca9f2c07dcd5b4e46"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.271956 13576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:47.283803 13576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:47.287225 13576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:47.287451 13576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:47.294857 13576 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.62:34617
I20260808 08:24:47.295149 13670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.62:34617 every 8 connection(s)
I20260808 08:24:47.295871 13576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/data/info.pb
I20260808 08:24:47.298780 13671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:47.299278 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13576
I20260808 08:24:47.299412 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-0/wal/instance
I20260808 08:24:47.299710 12340 external_mini_cluster.cc:1428] Setting key e20051601a7ba3161318cf5070c3c709
I20260808 08:24:47.303423 13671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.61@KRBTEST.COM" created.
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:47.319952 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.306323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:34617 (local address 127.0.0.1:59244)
0808 08:24:47.306828 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.306881 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.307727 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.308589 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.308594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.308917 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.309189 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.309201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.310735 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.310738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.311468 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.311477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.312162 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.312164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.312271 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.312954 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.312982 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.314759 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.316798 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.316807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.316816 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.317041 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.317375 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.317377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.317379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.317493 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.318159 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.318165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.318325 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.318495 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.318904 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":121,"thread_start_us":149,"threads_started":1}
I20260808 08:24:47.319952 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.306324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.12.13.62:34617)
0808 08:24:47.306959 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.306973 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.307026 (+ 53us) server_negotiation.cc:407] Connection header received
0808 08:24:47.308077 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.308087 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.308357 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.308574 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.309361 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.310617 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.311576 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.311956 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.312200 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.314919 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.314930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.314941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.314961 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.316665 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.317156 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.317169 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.317170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.317272 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.317591 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.317594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.317596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.318043 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.318191 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.318573 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.318853 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":478,"thread_start_us":150,"threads_started":1}
I20260808 08:24:47.322463 13671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:47.323500 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.323033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57500)
0808 08:24:47.323303 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.323425 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260808 08:24:47.323701 13598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
I20260808 08:24:47.324676 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.324396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57512)
0808 08:24:47.324555 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.324610 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260808 08:24:47.324915 13671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:37585: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab.
Entry for principal kudu/127.12.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab.
I20260808 08:24:47.331245 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:37585
--webserver_interface=127.12.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab
--principal=kudu/127.12.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:34617,127.12.13.61:37585,127.12.13.60:32867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:24:47.374176 13671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } attempt: 1
I20260808 08:24:47.375216 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.374816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57528)
0808 08:24:47.374998 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.375075 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 08:24:47.376060 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.375830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57532)
0808 08:24:47.375951 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.376002 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 08:24:47.376223 13671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:37585: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
I20260808 08:24:47.444377 13671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } attempt: 2
I20260808 08:24:47.445256 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.444943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57544)
0808 08:24:47.445099 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.445175 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 08:24:47.446127 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.445918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57552)
0808 08:24:47.446022 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.446070 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 08:24:47.446303 13671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:37585: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
W20260808 08:24:47.452296 13681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 08:24:47.452580 13681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:47.452654 13681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:47.452732 13681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:47.456890 13681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:47.456960 13681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:47.456981 13681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:47.456998 13681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:47.457050 13681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:47.462708 13681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:34617,127.12.13.61:37585,127.12.13.60:32867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.61.keytab
--principal=kudu/127.12.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.61:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:47.464052 13681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:47.465126 13681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:47.473937 13691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:47.473892 13690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:47.473839 13693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:47.474126 13681 server_base.cc:1034] running on GCE node
I20260808 08:24:47.474836 13681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:47.476049 13681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:47.477221 13681 hybrid_clock.cc:648] HybridClock initialized: now 1786177487477201 us; error 33 us; skew 500 ppm
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:47.480134 13681 init.cc:377] Logged in from keytab as kudu/127.12.13.61@KRBTEST.COM (short username kudu)
I20260808 08:24:47.481011 13681 webserver.cc:460] Webserver started at http://127.12.13.61:46505/ using document root <none> and password file <none>
I20260808 08:24:47.481559 13681 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:47.481619 13681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:47.481804 13681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:47.483935 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data/instance:
uuid: "1e003018e9b84103a9045d9bdaf1218e"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "8bf70ccd03de755510c23812ca40a4c5"
server_key_iv: "6b37ecc937153624a7705512bd2bdca5"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.484470 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal/instance:
uuid: "1e003018e9b84103a9045d9bdaf1218e"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "8bf70ccd03de755510c23812ca40a4c5"
server_key_iv: "6b37ecc937153624a7705512bd2bdca5"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.488262 13681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 08:24:47.490697 13701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:47.491942 13681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 08:24:47.492050 13681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal
uuid: "1e003018e9b84103a9045d9bdaf1218e"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "8bf70ccd03de755510c23812ca40a4c5"
server_key_iv: "6b37ecc937153624a7705512bd2bdca5"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.492136 13681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:47.537508 13671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } attempt: 3
I20260808 08:24:47.538558 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.538194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57558)
0808 08:24:47.538380 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.538462 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 08:24:47.539302 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.539087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57574)
0808 08:24:47.539191 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.539240 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 08:24:47.539450 13671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.61:37585: Network error: Client connection negotiation failed: client connection to 127.12.13.61:37585: connect: Connection refused (error 111)
I20260808 08:24:47.552644 13681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:47.556293 13681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:47.556533 13681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:47.564268 13681 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.61:37585
I20260808 08:24:47.564576 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.61:37585 every 8 connection(s)
I20260808 08:24:47.565426 13681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/data/info.pb
I20260808 08:24:47.568550 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:47.570176 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13681
I20260808 08:24:47.570358 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-1/wal/instance
I20260808 08:24:47.570730 12340 external_mini_cluster.cc:1428] Setting key a1dd26e729f45f7f3ae81238e06a8eef
I20260808 08:24:47.574226 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.13.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.13.60@KRBTEST.COM" created.
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:47.589771 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.576517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.12.13.62:34617)
0808 08:24:47.576685 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.576691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.577754 (+ 1063us) server_negotiation.cc:407] Connection header received
0808 08:24:47.578749 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.578753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.578816 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.578937 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.580148 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.580692 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.581955 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.582196 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.582318 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.585818 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.585821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.585823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.585848 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.587522 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.588088 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.588092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.588094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.588182 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.588548 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.588552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.588554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.588755 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.588874 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.589438 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.589576 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 08:24:47.590665 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.577043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:34617 (local address 127.0.0.1:59256)
0808 08:24:47.577570 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.577618 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.578527 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.579114 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.579121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.579486 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.579992 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.580007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.580836 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.580839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.581787 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.581798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.582365 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.582369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.582546 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.583206 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.583224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.585621 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.587671 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.587677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.587694 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.587975 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.588287 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.588290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.588292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.588445 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.588872 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.588880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.589160 (+ 280us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.589366 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.589706 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
I20260808 08:24:47.592110 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:47.601470 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.592646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57576)
0808 08:24:47.592831 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.592848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.592939 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.593532 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.593536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.593710 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.593853 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.593860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.594930 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.594936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.595473 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.595481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.596090 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.596093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.596186 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.596679 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.596699 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.597931 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.599876 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.599882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.599885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.600078 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.600383 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.600386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.600387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.600472 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.600940 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.600942 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.601038 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.601154 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.601292 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 08:24:47.601570 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.592782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.12.13.61:37585)
0808 08:24:47.593110 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.593120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.593139 (+ 19us) server_negotiation.cc:407] Connection header received
0808 08:24:47.593175 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.593179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.593392 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.593500 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.594010 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.594794 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.595606 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.595973 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.596091 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.598069 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.598086 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.598097 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.598118 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.599765 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.600191 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.600201 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.600203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.600281 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.600567 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.600570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.600571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.600825 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.600976 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.601267 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.601401 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab.
Entry for principal kudu/127.12.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab.
I20260808 08:24:47.604314 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:47.605084 12340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
/tmp/dist-test-taskpcRZlg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:32867
--webserver_interface=127.12.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab
--principal=kudu/127.12.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.13.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.13.62:34617,127.12.13.61:37585,127.12.13.60:32867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 08:24:47.605319 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.604924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52592)
0808 08:24:47.605070 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.605153 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 08:24:47.605544 13707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
I20260808 08:24:47.606688 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.606430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52606)
0808 08:24:47.606557 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.606616 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 08:24:47.606976 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:32867: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
I20260808 08:24:47.656370 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } attempt: 1
I20260808 08:24:47.657405 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.657067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52616)
0808 08:24:47.657238 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.657317 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 08:24:47.658241 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.658023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52628)
0808 08:24:47.658133 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.658187 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 08:24:47.658383 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:32867: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
I20260808 08:24:47.682639 13671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } attempt: 4
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:47.693481 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.683258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57582)
0808 08:24:47.683447 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.683481 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.683567 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.683882 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.683884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.684070 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.684173 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.684179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.686114 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.686117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.687055 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.687066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.687705 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.687708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.687808 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.688384 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.688397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.689940 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.691927 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.691929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.691931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.692130 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.692422 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.692424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.692425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.692505 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.692995 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.692997 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.693089 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.693179 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.693345 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 08:24:47.693578 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.683351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.12.13.61:37585)
0808 08:24:47.683499 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.683503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.683554 (+ 51us) server_negotiation.cc:407] Connection header received
0808 08:24:47.683674 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.683678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.683744 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.683833 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.684316 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.685987 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.687196 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.687575 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.687683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.690079 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.690083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.690086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.690115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.691813 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.692230 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.692234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.692236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.692325 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.692604 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.692609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.692612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.692882 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.692986 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.693275 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.693414 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 08:24:47.694622 13671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:47.695456 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.695153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52644)
0808 08:24:47.695324 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.695393 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 08:24:47.696456 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.696219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52648)
0808 08:24:47.696338 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.696400 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 08:24:47.696671 13671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:32867: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
I20260808 08:24:47.726564 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } attempt: 2
I20260808 08:24:47.727494 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.727155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52658)
0808 08:24:47.727321 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.727408 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 08:24:47.728274 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.728051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52668)
0808 08:24:47.728169 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.728222 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 08:24:47.728422 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:32867: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
W20260808 08:24:47.733423 13798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 08:24:47.733755 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 08:24:47.733819 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 08:24:47.733916 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 08:24:47.738037 13798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 08:24:47.738121 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 08:24:47.738157 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 08:24:47.738191 13798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 08:24:47.738226 13798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 08:24:47.743948 13798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.13.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.13.62:34617,127.12.13.61:37585,127.12.13.60:32867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/krb5kdc/kudu_127.12.13.60.keytab
--principal=kudu/127.12.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.13.60:32867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 07:34:22 UTC on ccycloud.quasar-bqoudn.root.comops.site
build id 4574264
I20260808 08:24:47.745198 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 08:24:47.746383 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 08:24:47.755682 13810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 08:24:47.755702 13808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:47.755861 13671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } attempt: 1
W20260808 08:24:47.755682 13805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 08:24:47.756445 13798 server_base.cc:1034] running on GCE node
I20260808 08:24:47.756783 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.756441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52670)
0808 08:24:47.756637 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.756701 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 08:24:47.757164 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 08:24:47.757630 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.757395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52674)
0808 08:24:47.757535 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.757582 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 08:24:47.757843 13671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.13.60:32867: Network error: Client connection negotiation failed: client connection to 127.12.13.60:32867: connect: Connection refused (error 111)
I20260808 08:24:47.758634 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 08:24:47.759814 13798 hybrid_clock.cc:648] HybridClock initialized: now 1786177487759793 us; error 27 us; skew 500 ppm
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 08:24:47.762760 13798 init.cc:377] Logged in from keytab as kudu/127.12.13.60@KRBTEST.COM (short username kudu)
I20260808 08:24:47.763734 13798 webserver.cc:460] Webserver started at http://127.12.13.60:32815/ using document root <none> and password file <none>
I20260808 08:24:47.764272 13798 fs_manager.cc:359] Metadata directory not provided
I20260808 08:24:47.764331 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 08:24:47.764505 13798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 08:24:47.766707 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data/instance:
uuid: "a61b0177f54a43cba46ade21f4523e7a"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "6fcaab5db998ff3de38078cab4673eac"
server_key_iv: "990e09de5624e718005df5e103b7d75f"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.767269 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal/instance:
uuid: "a61b0177f54a43cba46ade21f4523e7a"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "6fcaab5db998ff3de38078cab4673eac"
server_key_iv: "990e09de5624e718005df5e103b7d75f"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.771227 13798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 08:24:47.773939 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 08:24:47.775518 13798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 08:24:47.775671 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data,/tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal
uuid: "a61b0177f54a43cba46ade21f4523e7a"
format_stamp: "Formatted at 2026-08-08 08:24:47 on dist-test-slave-6ntk"
server_key: "6fcaab5db998ff3de38078cab4673eac"
server_key_iv: "990e09de5624e718005df5e103b7d75f"
server_key_version: "encryptionkey@0"
I20260808 08:24:47.775799 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 08:24:47.802030 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 08:24:47.805485 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 08:24:47.805713 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 08:24:47.813592 13798 rpc_server.cc:307] RPC server started. Bound to: 127.12.13.60:32867
I20260808 08:24:47.813951 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.13.60:32867 every 8 connection(s)
I20260808 08:24:47.814755 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/data/info.pb
I20260808 08:24:47.817691 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 08:24:47.819592 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } attempt: 3
I20260808 08:24:47.822132 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:47.824203 12340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu as pid 13798
I20260808 08:24:47.824337 12340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpcRZlg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786177179845590-12340-0/minicluster-data/master-2/wal/instance
I20260808 08:24:47.824712 12340 external_mini_cluster.cc:1428] Setting key 45e0817793b2d517c9aa52e09e4d1486
I20260808 08:24:47.825030 13671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } attempt: 2
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.61@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.62@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:47.835377 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.820164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52680)
0808 08:24:47.820329 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.820355 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.820464 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.822883 (+ 2419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.822887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.823106 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.823227 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.823235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.825059 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.825062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.825913 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.825927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.827231 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.827235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.827327 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.827899 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.827918 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.829530 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.832399 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.832403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.832406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.832707 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.833208 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.833213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.833216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.833319 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.834742 (+ 1423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.834745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.834859 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.834965 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.835153 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 08:24:47.836659 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.825403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52710)
0808 08:24:47.825594 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.825609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.825684 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.826242 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.826245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.826421 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.826523 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.826530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.828034 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.828037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.828828 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.828838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.829460 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.829464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.829663 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.830230 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.830242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.831823 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.834670 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.834673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.834676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.835158 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.835470 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.835474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.835476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.835570 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.835974 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.835980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.836102 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.836289 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.836459 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":94}
I20260808 08:24:47.837597 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.825564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.12.13.60:32867)
0808 08:24:47.825958 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.825968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.825982 (+ 14us) server_negotiation.cc:407] Connection header received
0808 08:24:47.826015 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.826029 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.826075 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.826637 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.826739 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.827871 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.828992 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.829333 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.829506 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.832009 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.832012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.832015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.832044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.834543 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.835252 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.835256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.835258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.835345 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.835679 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.835681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.835682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.835854 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.835964 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.837005 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.837123 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":283,"thread_start_us":71,"threads_started":1}
I20260808 08:24:47.837576 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.820878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.12.13.60:32867)
0808 08:24:47.821302 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.821324 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.821400 (+ 76us) server_negotiation.cc:407] Connection header received
0808 08:24:47.822144 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.822164 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.822562 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.822914 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.823383 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.824909 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.826060 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.826844 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.827025 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.829723 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.829730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.829745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.829777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.832243 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.832898 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.832904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.832911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.833076 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.833501 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.833509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.833512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.834629 (+ 1117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.834791 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.835175 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.836189 (+ 1014us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260808 08:24:47.838469 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.823997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.12.13.62:34617)
0808 08:24:47.824137 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.824142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.824585 (+ 443us) server_negotiation.cc:407] Connection header received
0808 08:24:47.824820 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.824823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.824888 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.824987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.826659 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.827429 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.829063 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.829379 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.829462 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.834103 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.834107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.834110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.834137 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.835926 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.836556 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.836560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.836562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.836652 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.837287 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.837292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.837294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.837554 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.837627 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.838198 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.838319 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":57}
I20260808 08:24:47.838524 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.823904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.62:34617 (local address 127.0.0.1:59270)
0808 08:24:47.824432 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.824491 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.824687 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.825710 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.825719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.826323 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.826516 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.826525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.827603 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.827607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.828934 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.828948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.829723 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.829729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.830745 (+ 1016us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.831633 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.831648 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.833950 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.836047 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.836060 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.836070 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.836440 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.836994 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.836999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.837002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.837102 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.837713 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.837721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.837850 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.838096 (+ 246us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.838297 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":337,"thread_start_us":165,"threads_started":1}
I20260808 08:24:47.839958 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.825321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.12.13.60:32867)
0808 08:24:47.825820 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.825828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.826138 (+ 310us) server_negotiation.cc:407] Connection header received
0808 08:24:47.826305 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.826308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.826351 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.826408 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.827489 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.828721 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.830024 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.830437 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.830570 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.833428 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.833431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.833434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.833462 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.837512 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.838082 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.838088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.838090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.838175 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.838492 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.838495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.838497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.838782 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.838973 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.839640 (+ 667us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.839762 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":384,"server-negotiator.queue_time_us":444,"thread_start_us":96,"threads_started":1}
I20260808 08:24:47.840404 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:47.842772 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: Bootstrap starting.
I20260808 08:24:47.842772 13671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad: Bootstrap starting.
I20260808 08:24:47.845501 13671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:47.845520 13785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:47.846482 13671 log.cc:824] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:47.846895 13785 log.cc:824] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: Log is configured to *not* fsync() on all Append() calls
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:47.849038 13671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad: No bootstrap required, opened a new log
I20260808 08:24:47.849038 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: No bootstrap required, opened a new log
I20260808 08:24:47.851783 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.842037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.12.13.61:37585)
0808 08:24:47.842178 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.842182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.842197 (+ 15us) server_negotiation.cc:407] Connection header received
0808 08:24:47.842234 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.842236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.842281 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.842372 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.842947 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.843668 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.844782 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.845088 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.845203 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.847881 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.847887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.847894 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.847929 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.849757 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.850299 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.850303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.850304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.850461 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.850774 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.850777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.850779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.851029 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.851115 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.851492 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.851622 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 08:24:47.852319 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.840962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.61:37585 (local address 127.0.0.1:57590)
0808 08:24:47.841100 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.841115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.841193 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.842496 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.842500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.842677 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.842802 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.842811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.843816 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.843819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.844636 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.844648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.845229 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.845233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.845353 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.846023 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.846037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.847695 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.849910 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.849922 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.849924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.850186 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.850584 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.850587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.850589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.850676 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.851193 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.851195 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.851293 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.851935 (+ 642us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.852112 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 08:24:47.853166 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
I20260808 08:24:47.853461 13785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.853699 13785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e003018e9b84103a9045d9bdaf1218e, State: Initialized, Role: FOLLOWER
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.62@KRBTEST.COM
I20260808 08:24:47.854266 13785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.854694 13671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.854910 13671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3bb418cae14ae7ad41f9b0837c3fad, State: Initialized, Role: FOLLOWER
I20260808 08:24:47.855213 13671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.855831 13785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:47.856500 13671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:47.858848 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.842961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.12.13.62:34617)
0808 08:24:47.843296 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.843300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.847140 (+ 3840us) server_negotiation.cc:407] Connection header received
0808 08:24:47.847404 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.847409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.847468 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.847553 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.849203 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.850021 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.851146 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.851467 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.851557 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.854614 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.854618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.854621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.854651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.856373 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.857356 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.857363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.857364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.857458 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.857768 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.857771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.857773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.858012 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.858178 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.858569 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.858690 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
W20260808 08:24:47.862916 13939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:47.863006 13939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:47.866178 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.866308 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.13.61@KRBTEST.COM
I20260808 08:24:47.865304 13576 master_runner.cc:186] Error getting master registration for 127.12.13.60:32867: OK, instance_id { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" instance_seqno: 1786177487811472 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 08:24:47.868840 13576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 08:24:47.870124 13930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:47.870303 13930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:47.870487 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.870621 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:47.872764 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.860899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.12.13.61:37585)
0808 08:24:47.861653 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.861658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.861674 (+ 16us) server_negotiation.cc:407] Connection header received
0808 08:24:47.861709 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.861712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.861754 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.861827 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.862896 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.863595 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.864986 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.865348 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.865470 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.868548 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.868552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.868555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.868600 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.870314 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.870827 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.870831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.870833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.870938 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.871249 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.871252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.871254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.871582 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.871790 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.872478 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.872599 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687}
Aug 08 08:24:47 dist-test-slave-6ntk krb5kdc[13559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786177487, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.13.60@KRBTEST.COM for kudu/127.12.13.60@KRBTEST.COM
I20260808 08:24:47.870107 13681 master_runner.cc:186] Error getting master registration for 127.12.13.60:32867: OK, instance_id { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" instance_seqno: 1786177487811472 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 08:24:47.875547 13681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 08:24:47.876576 12340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 08:24:47.877907 13634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280
I20260808 08:24:47.878225 13634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 08:24:47.878350 13634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:47.879096 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.854017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.12.13.60:32867)
0808 08:24:47.854138 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.854143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.854692 (+ 549us) server_negotiation.cc:407] Connection header received
0808 08:24:47.855179 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.855182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.855241 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.855305 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.856121 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.856925 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.858332 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.858659 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.858736 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.875043 (+ 16307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.875048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.875051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.875078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.876978 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.877517 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.877520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.877522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.877609 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.877897 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.877900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.877902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.878155 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.878252 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.878753 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.878889 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 08:24:47.879822 13634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.879905 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.853709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.13.60:32867 (local address 127.0.0.1:52716)
0808 08:24:47.854631 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0808 08:24:47.854646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 08:24:47.854748 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 08:24:47.855448 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 08:24:47.855691 (+ 243us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 08:24:47.855889 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 08:24:47.855993 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.856002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.857412 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.857417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.858223 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 08:24:47.858235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.858875 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.858881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 08:24:47.859891 (+ 1010us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.866015 (+ 6124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 08:24:47.866031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 08:24:47.874877 (+ 8846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 08:24:47.877109 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.877113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.877116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.877404 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.877704 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 08:24:47.877707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 08:24:47.877709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 08:24:47.877784 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 08:24:47.878474 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 08:24:47.878478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 08:24:47.878576 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 08:24:47.879594 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0808 08:24:47.879747 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20260808 08:24:47.881304 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1e003018e9b84103a9045d9bdaf1218e"
I20260808 08:24:47.881579 13744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:47.882964 13744 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3bb418cae14ae7ad41f9b0837c3fad in term 1.
I20260808 08:24:47.883399 13598 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e003018e9b84103a9045d9bdaf1218e, 6c3bb418cae14ae7ad41f9b0837c3fad; no voters:
I20260808 08:24:47.883579 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: Bootstrap starting.
I20260808 08:24:47.883726 13908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 1 FOLLOWER]: Leader election won for term 1
I20260808 08:24:47.883983 13908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [term 1 LEADER]: Becoming Leader. State: Replica: 6c3bb418cae14ae7ad41f9b0837c3fad, State: Running, Role: LEADER
I20260808 08:24:47.884339 13908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.886232 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3bb418cae14ae7ad41f9b0837c3fad. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.886345 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:47.886623 13893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: Neither blocks nor log segments found. Creating new log.
I20260808 08:24:47.886675 13634 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [CANDIDATE]: Term 1 election: Requested vote from peers 1e003018e9b84103a9045d9bdaf1218e (127.12.13.61:37585), a61b0177f54a43cba46ade21f4523e7a (127.12.13.60:32867)
I20260808 08:24:47.887563 13893 log.cc:824] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: Log is configured to *not* fsync() on all Append() calls
I20260808 08:24:47.890281 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 08:24:47.895237 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 08:24:47.900622 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.889235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.12.13.60:32867)
0808 08:24:47.889349 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.889354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.889928 (+ 574us) server_negotiation.cc:407] Connection header received
0808 08:24:47.890126 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.890129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.890176 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.890227 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.891241 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.892036 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.893232 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.893579 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.893652 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.896387 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.896392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.896395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.896422 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.898124 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.898883 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.898887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.898889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.898980 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.899455 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.899459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.899461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.899678 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.899780 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.900322 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.900453 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 08:24:47.912015 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: No bootstrap required, opened a new log
I20260808 08:24:47.916103 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.891226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.12.13.61:37585)
0808 08:24:47.891366 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.891370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.898148 (+ 6778us) server_negotiation.cc:407] Connection header received
0808 08:24:47.898415 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.898419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.898473 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.898524 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.901189 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.901960 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.904279 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.904655 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.904758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.910419 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.910424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.910427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.910458 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.912507 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.914175 (+ 1668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.914179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.914181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.914271 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.914573 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.914576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.914578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.914811 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.914963 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.915751 (+ 788us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.915925 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 08:24:47.918100 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:47.889510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.12.13.62:34617)
0808 08:24:47.889620 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:47.889624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:47.897785 (+ 8161us) server_negotiation.cc:407] Connection header received
0808 08:24:47.899379 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:47.899383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:47.899433 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:47.899486 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:47.900280 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.901052 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.903301 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:47.903660 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:47.903755 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:47.910835 (+ 7080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 08:24:47.910840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 08:24:47.910858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 08:24:47.910890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 08:24:47.913279 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.915298 (+ 2019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.915302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.915304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.915406 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 08:24:47.917008 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 08:24:47.917012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 08:24:47.917014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 08:24:47.917243 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 08:24:47.917353 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:47.917779 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 08:24:47.917945 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 08:24:47.918497 13893 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.918908 13893 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61b0177f54a43cba46ade21f4523e7a, State: Initialized, Role: FOLLOWER
I20260808 08:24:47.919504 13893 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } }
I20260808 08:24:47.921397 13744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [term 1 FOLLOWER]: Refusing update from remote peer 6c3bb418cae14ae7ad41f9b0837c3fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:47.921998 13855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a61b0177f54a43cba46ade21f4523e7a"
I20260808 08:24:47.922120 13908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:47.922199 13854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [term 0 FOLLOWER]: Advancing to term 1
I20260808 08:24:47.922299 13893 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 08:24:47.923833 13854 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [term 1 FOLLOWER]: Refusing update from remote peer 6c3bb418cae14ae7ad41f9b0837c3fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 08:24:47.925681 13855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3bb418cae14ae7ad41f9b0837c3fad in term 1.
I20260808 08:24:47.927469 13908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 08:24:47.930346 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3bb418cae14ae7ad41f9b0837c3fad. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.930477 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:47.926966 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.931202 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:47.931542 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3bb418cae14ae7ad41f9b0837c3fad. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.931650 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:47.934491 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.934626 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:47.935432 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.935545 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:47.935804 13958 mvcc.cc:205] Tried to move back new op lower bound from 7316182990465769472 to 7316182990377779200. Current Snapshot: MvccSnapshot[applied={T|T < 7316182990465769472}]
I20260808 08:24:47.938908 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.939050 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad [sys.catalog]: This master's current role is: LEADER
I20260808 08:24:47.940135 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3bb418cae14ae7ad41f9b0837c3fad" member_type: VOTER last_known_addr { host: "127.12.13.62" port: 34617 } } peers { permanent_uuid: "1e003018e9b84103a9045d9bdaf1218e" member_type: VOTER last_known_addr { host: "127.12.13.61" port: 37585 } } peers { permanent_uuid: "a61b0177f54a43cba46ade21f4523e7a" member_type: VOTER last_known_addr { host: "127.12.13.60" port: 32867 } } }
I20260808 08:24:47.940281 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 08:24:47.941984 13985 mvcc.cc:205] Tried to move back new op lower bound from 7316182990465769472 to 7316182990377779200. Current Snapshot: MvccSnapshot[applied={T|T < 7316182990465769472}]
W20260808 08:24:47.942560 13979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 08:24:47.942641 13979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 08:24:47.947144 13944 catalog_manager.cc:1347] Generated new cluster ID: 914d4456c71d4b72a7460895d640da26
I20260808 08:24:47.947229 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 08:24:47.966010 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 08:24:47.967448 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260808 08:24:47.978334 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3bb418cae14ae7ad41f9b0837c3fad: Generated new TSK 0
I20260808 08:24:47.979310 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 08:24:48.017282 13898 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.013806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.12.13.60:32867)
0808 08:24:48.013982 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.013987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.016091 (+ 2104us) server_negotiation.cc:407] Connection header received
0808 08:24:48.016303 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.016306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.016361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.016416 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.017120 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52736: BlockingRecv error: recv got EOF from 127.0.0.1:52736 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260808 08:24:48.018909 13676 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.014761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.12.13.62:34617)
0808 08:24:48.014893 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.014897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.017842 (+ 2945us) server_negotiation.cc:407] Connection header received
0808 08:24:48.018096 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.018100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.018157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.018212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.018774 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59298: BlockingRecv error: recv got EOF from 127.0.0.1:59298 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 08:24:48.020408 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.015455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.12.13.61:37585)
0808 08:24:48.015576 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.015580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.019369 (+ 3789us) server_negotiation.cc:407] Connection header received
0808 08:24:48.019642 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.019646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.019704 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.019753 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.020279 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57620: BlockingRecv error: recv got EOF from 127.0.0.1:57620 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 08:24:48.025499 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.023811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.12.13.61:37585)
0808 08:24:48.023952 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.023956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.024438 (+ 482us) server_negotiation.cc:407] Connection header received
0808 08:24:48.024678 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.024682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.024770 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.024823 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.025383 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57636: BlockingRecv error: recv got EOF from 127.0.0.1:57636 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 08:24:48.031052 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.029351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.12.13.61:37585)
0808 08:24:48.029458 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.029462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.030072 (+ 610us) server_negotiation.cc:407] Connection header received
0808 08:24:48.030288 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.030291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.030348 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.030401 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.030941 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57648: BlockingRecv error: recv got EOF from 127.0.0.1:57648 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 08:24:48.037313 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.035608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.12.13.61:37585)
0808 08:24:48.035721 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.035727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.036276 (+ 549us) server_negotiation.cc:407] Connection header received
0808 08:24:48.036503 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.036507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.036565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.036622 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.037189 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57658: BlockingRecv error: recv got EOF from 127.0.0.1:57658 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 08:24:48.045744 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.043827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.12.13.61:37585)
0808 08:24:48.044006 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.044010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.044633 (+ 623us) server_negotiation.cc:407] Connection header received
0808 08:24:48.044886 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.044890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.044952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.045009 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.045618 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57674: BlockingRecv error: recv got EOF from 127.0.0.1:57674 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260808 08:24:48.058352 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.056583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.12.13.61:37585)
0808 08:24:48.056712 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.056717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.057304 (+ 587us) server_negotiation.cc:407] Connection header received
0808 08:24:48.057515 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.057518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.057570 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.057622 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.058227 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57678: BlockingRecv error: recv got EOF from 127.0.0.1:57678 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 08:24:48.079306 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.077391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.12.13.61:37585)
0808 08:24:48.077520 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.077525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.078181 (+ 656us) server_negotiation.cc:407] Connection header received
0808 08:24:48.078395 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.078399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.078456 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.078509 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.079183 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57692: BlockingRecv error: recv got EOF from 127.0.0.1:57692 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 08:24:48.115629 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.114228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.12.13.61:37585)
0808 08:24:48.114374 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.114379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.114621 (+ 242us) server_negotiation.cc:407] Connection header received
0808 08:24:48.114804 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.114807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.114868 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.114972 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.115481 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57704: BlockingRecv error: recv got EOF from 127.0.0.1:57704 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 08:24:48.183497 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.182008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.12.13.61:37585)
0808 08:24:48.182187 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.182194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.182334 (+ 140us) server_negotiation.cc:407] Connection header received
0808 08:24:48.182539 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.182545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.182621 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.182741 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.183345 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57710: BlockingRecv error: recv got EOF from 127.0.0.1:57710 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 08:24:48.314987 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.313686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.12.13.61:37585)
0808 08:24:48.313856 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.313859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.314064 (+ 205us) server_negotiation.cc:407] Connection header received
0808 08:24:48.314198 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.314201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.314255 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.314346 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.314871 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57714: BlockingRecv error: recv got EOF from 127.0.0.1:57714 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260808 08:24:48.574749 13795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 08:24:48.573466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.12.13.61:37585)
0808 08:24:48.573636 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:48.573640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:48.573771 (+ 131us) server_negotiation.cc:407] Connection header received
0808 08:24:48.573985 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:48.573988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:48.574042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:48.574109 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 08:24:48.574622 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57724: BlockingRecv error: recv got EOF from 127.0.0.1:57724 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 08:24:48.871796 13930 catalog_manager.cc:1259] Loaded cluster ID: 914d4456c71d4b72a7460895d640da26
I20260808 08:24:48.871901 13930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: loading cluster ID for follower catalog manager: success
I20260808 08:24:48.873695 13930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: acquiring CA information for follower catalog manager: success
I20260808 08:24:48.875002 13930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e003018e9b84103a9045d9bdaf1218e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:24:48.944319 13979 catalog_manager.cc:1259] Loaded cluster ID: 914d4456c71d4b72a7460895d640da26
I20260808 08:24:48.944450 13979 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: loading cluster ID for follower catalog manager: success
I20260808 08:24:48.946677 13979 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: acquiring CA information for follower catalog manager: success
I20260808 08:24:48.948203 13979 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a61b0177f54a43cba46ade21f4523e7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 08:24:49.093096 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:49.089073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.12.13.61:37585)
0808 08:24:49.089351 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:49.089355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:49.089378 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:24:49.089544 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:49.089548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:49.089691 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:49.089808 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:49.090179 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.090708 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.091571 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.091798 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.091903 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:49.092181 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:49.092687 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:49.092787 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:49.092906 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 08:24:49.092960 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
I20260808 08:24:49.100463 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:49.096565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.12.13.60:32867)
0808 08:24:49.096843 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:49.096859 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:49.096880 (+ 21us) server_negotiation.cc:407] Connection header received
0808 08:24:49.096965 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:49.096967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:49.097088 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:49.097221 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:49.097461 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.098056 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.098898 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.099127 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.099253 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:49.099565 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:49.100019 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:49.100164 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:49.100226 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 08:24:49.100286 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260808 08:24:49.108953 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:49.103971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.12.13.60:32867)
0808 08:24:49.104263 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:49.104267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:49.104354 (+ 87us) server_negotiation.cc:407] Connection header received
0808 08:24:49.104560 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:49.104563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:49.104617 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:49.104707 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:49.104948 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.105718 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.107336 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.107718 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.107807 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:49.108279 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:49.108387 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:49.108453 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:49.108684 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 08:24:49.108748 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260808 08:24:49.109067 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:49.104392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.12.13.61:37585)
0808 08:24:49.104526 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:49.104531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:49.104818 (+ 287us) server_negotiation.cc:407] Connection header received
0808 08:24:49.105031 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:49.105035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:49.105091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:49.105164 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:49.105522 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.106294 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.107654 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.108028 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.108141 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:49.108570 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:49.108655 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:49.108720 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:49.108866 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 08:24:49.108925 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 08:24:49.109215 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 08:24:49.103729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.12.13.62:34617)
0808 08:24:49.104214 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 08:24:49.104221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 08:24:49.104244 (+ 23us) server_negotiation.cc:407] Connection header received
0808 08:24:49.104371 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 08:24:49.104375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 08:24:49.104585 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 08:24:49.104739 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 08:24:49.105290 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.106173 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.107383 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 08:24:49.107672 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 08:24:49.107768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 08:24:49.108225 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 08:24:49.108580 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 08:24:49.108670 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 08:24:49.109007 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 08:24:49.109066 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":120,"threads_started":1}
I20260808 08:24:49.112877 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13576
I20260808 08:24:49.131181 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13681
I20260808 08:24:49.150398 12340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpcRZlg/build/debug/bin/kudu with pid 13798
2026-08-08T08:24:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3543 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7092 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (309261 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST